IndexSplitter.split commits the destination index without fsyncing the segment files it copied

IndexSplitter.split (lucene/misc, also the command line IndexSplitter <srcDir> <destDir> <segments>+) copies every file of the chosen segments into the destination with Files.copy and then publishes a commit there with SegmentInfos.commit. That commit fsyncs only the new segments_N and the directory; nothing fsyncs the copied files. So the destination's commit can be on stable storage while the segment files it names are only in the OS page cache, which breaks the rule every other Lucene commit path follows (IndexWriter.commit and the NRT replicator sync the referenced files first). After an OS crash or power loss before writeback, the durable commit can name empty or damaged files, and then the destination does not open: in Lucene's own crash model DirectoryReader.open fails with CorruptSegmentInfoException. The source index is untouched. Present since the tool was added in 3.0.0 (2009), through 10.6.0 and main; the fix is to sync the copied files before the commit.

1. It opens the destination as an FSDirectory and builds a new SegmentInfos (IndexSplitter.java:109-140).
2. For each chosen segment it copies every file of SegmentCommitInfo.files() with Files.copy(srcFile, destFile) (:141-147, the copy at :146). Files.copy never fsyncs.
3. It calls destInfos.changed() and destInfos.commit(destFSDir) (:149-150). There is no sync call anywhere in the class.

• commit(dir) is prepareCommit(dir) then finishCommit(dir) (SegmentInfos.java:1031-1034; javadoc "Writes and syncs to the Directory dir").
• prepareCommit: dir.syncMetaData() (:970), then write(dir), which creates pending_segments_N and fsyncs only that file (:620).
• finishCommit: renames it to segments_N (:1005) and fsyncs the directory again (:1007).

IndexWriter.startCommit calls prepareCommit (IndexWriter.java:5974), then directory.sync(toSync.files(false)) (:5992-5993), and only later finishCommit. So segments_N becomes durable only after every file it names is. ReplicaNode does the same ("fsync all index files we are now referencing", ReplicaNode.java:364-373). split() skips that middle step.

copy _0.cfe, copy _0.si, copy _0.cfs; fsync the directory; write and fsync pending_segments_1; rename to segments_1; fsync the directory. The commit names _0.cfe, _0.cfs, _0.si and segments_1; only segments_1 was fsynced. In a post-crash image with the 3 copies left empty, zero-filled or deleted (three of the six damage modes of MockDirectoryWrapper.crash; the probe builds these images itself), DirectoryReader.open fails with CorruptSegmentInfoException: segment info file: _0.si cannot be read - it may be missing or corrupt. Before any crash the destination opens with maxDoc 1.

• Who. Operators who run the @lucene.experimental IndexSplitter (IndexSplitter.java:44) from code or the command line. Normal indexing is not affected: IndexWriter.commit and close() fsync every referenced file in every version run, and IndexWriter.addIndexes(Directory) plus commit leaves no referenced file unsynced.
• Trigger. An OS crash, kernel panic, power loss or hard reset of the machine or VM after split() returns and before the kernel writes the copies back. On Linux, dirty data is written back once it is about 30 s old by default (vm.dirty_expire_centisecs = 3000 on the test host). Ending the process is not enough: after Runtime.halt(0) right after split(), 5 of 5 destinations opened intact in a fresh JVM, since the copies stay in the OS page cache; a kill -9 or JVM crash ends the process the same way (not run). How likely a real crash is to catch copies not yet written back depends on the file system, memory pressure and timing.
• Outcome. The destination's newest (and only) commit names lost files, so it does not open. The tested crash states fail with an exception at open; no silent wrong results were seen.
• Severity. Usually recoverable: split() never modifies the source, so the split can be re-run. Data is lost only when the source no longer has the segment's files either, for example in the follow-up the tool supports: split, then IndexSplitter <src> -d <seg> (usage text IndexSplitter.java:58, code :100-107; upstream TestIndexSplitter.java:97-98 splits and then removes the segment this way), then an IndexWriter on the source deletes the unreferenced files, then the crash hits before the copies are on disk.
• File-system dependence (not tested). On ext4 with delayed allocation, a new file whose data was never written back can come back empty after a crash; where Files.copy clones extents (copy_file_range on XFS or btrfs), the copy may survive. Only the crash model was used, on ext4; no real power cut was induced.
• Likelihood. Rare: the experimental tool plus a machine-level crash in a short window; nobody has reported it since 2009.

Public API only: IndexSplitter, SegmentInfos.readLatestCommit and the test-framework mockfile classes. split() takes a Path and opens its own FSDirectory, so MockDirectoryWrapper cannot see its writes; the test wraps the destination path in a FilterFileSystemProvider whose FileChannel.force records the file name, and @SuppressFileSystems("*") keeps the framework's own mock file systems out. Two limits: only the destination is wrapped, so the test's Files.copy crosses file-system providers and copies through streams rather than the JDK's native same-file-system copy (neither fsyncs: the independent reproducer's same-provider tracker and its no-wrapper flight-recorder check see no force on the copies either); and it checks the commit's segment files (files(false)), not segments_1 itself. Seeds DEADBEEF and 5EED1234 (5 iterations) fail the same way; the seed sets only the framework's randomization (codec, locale and similar).

checkGoogleJavaFormat rejects its one unused import, java.nio.file.FileSystem; that is the only format issue, and the version without it was run too (the failure is then reported at line 72 instead of 73).

What a real crash does next is not part of the test. In Lucene's crash model (the same split, with the 3 copies left empty, zero-filled or deleted), opening the destination fails with CorruptSegmentInfoException on _0.si on main, and with CorruptIndexException on 10.5.2 and 9.12.3.

# gradle, main 645d4fe14a20, seed DEADBEEF, assertions on (assertions off: the same)
TestIndexSplitterFsync > testSplitFsyncsCopiedFiles FAILED
    java.lang.AssertionError: segment files of the split's commit never fsynced: [_0.cfe, _0.cfs, _0.si]
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:C2ADA7351EE6702E]:0)
        at junit@4.13.2/org.junit.Assert.fail(Assert.java:89)
        at junit@4.13.2/org.junit.Assert.assertTrue(Assert.java:42)
        at org.apache.lucene.misc.index.TestIndexSplitterFsync.testSplitFsyncsCopiedFiles(TestIndexSplitterFsync.java:73)

# repro harness, the same message: main -ea, main -da, 10.6.0 -ea, 10.6.0 -da, 10.5.2 -ea, 9.12.3 -ea (1 run, 1 failed each)
# with the fix: the test passes (gradle, assertions on and off)

• The commit contract. Directory.java:105-108: sync "Ensures that any writes to these files are moved to stable storage (made durable)." and "Lucene uses this to properly commit changes to the index, to prevent a machine/OS crash from corrupting the index." IndexWriter.java:4188-4191: commit "syncs all referenced index files, such that a reader will see the changes and the index updates will survive an OS or machine crash or power loss."
• Lucene's crash model. MockDirectoryWrapper.java:512: "Simulates a crash of OS or machine by overwriting unsynced files."
• Every other publisher syncs first. IndexWriter.startCommit (IndexWriter.java:5992-5993) and ReplicaNode (:364-373). Among the non-test SegmentInfos.commit callers, split() is the only one that writes or copies segment data right before its commit: IndexSplitter.remove and CheckIndex's exorcise only rewrite the commit over files already in the index, and IndexRearranger commits segments that its own IndexWriter has just committed and closed (IndexRearranger.java:126-130, :170-173).
• No carve-out. The class javadoc (IndexSplitter.java:35-44) lists the tool's limits (whole segments only, FSDirectory only, segments are easy to remove by accident), not durability. The IndexWriter.java:1513-1516 note that offline splitting tools are used "at your own risk!" is about breaking up atomically added document blocks. Upstream TestIndexSplitter checks only the split result in-process.
• History: an omission. LUCENE-1959 (#3034, 2009) never mentions fsync, sync, crash or durability (body and all comments searched); the tool went in as "Progress not perfection!". None of the 84 commits that touch the file (all branches, its four paths, the introducing commit included) added or removed a sync.

The strongest case for "intended": IndexSplitter is an experimental command-line tool documented as "basic file-level copying" that promises no durability; SegmentInfos.commit promises only to write and sync the segments file, so callers must sync their own files; the crash state is synthetic.

Why it fails: the tool publishes a real Lucene commit point, with both directory fsyncs, and nothing says this commit is exempt from Lucene's commit rule; @lucene.experimental is about API stability. The narrow reading of SegmentInfos.commit only decides where the fix goes (in IndexSplitter, as IndexWriter and ReplicaNode do it). The crash state is Lucene's own model, and the repro asserts only the contract. Nothing found (javadoc, test, CHANGES, issue or list thread) calls non-durable split output intended; every verifier rejected the intended reading, and those that rated its strength called it moderate. Maintainers may still treat it as a small robustness fix to an experimental tool.

One fix was built and run through the module's tests; the alternatives were not run.

In IndexSplitter.split, collect every copied name and call destFSDir.sync(copied) before destInfos.commit(destFSDir) (shown condensed; the patch also adds the ArrayList and List imports). FSDirectory.sync fsyncs each named file; the commit's own directory fsyncs then make the names durable. The copied names are all of SegmentCommitInfo.files(): the segment's files plus any live-docs, field-infos and doc-values update files.

    List<String> copied = new ArrayList<>();
    for (String n : segs) {
      // ... build the destination's SegmentCommitInfo, unchanged ...
      for (final String srcName : files) {
        // ... resolve srcFile and destFile, unchanged ...
        Files.copy(srcFile, destFile);
        copied.add(srcName);
      }
    }
    // the commit below must only reference files whose bytes are on stable storage
    destFSDir.sync(copied);
    destInfos.changed();
    destInfos.commit(destFSDir);

Trade-offs

One fsync per copied file, once per split, in an offline tool. No behaviour change otherwise.

What the checks validated

• Scratch worktree of main 645d4fe14a20 (removed afterwards), seed DEADBEEF: stock, the test fails through gradle (assertions on and off); with A it passes, together with upstream TestIndexSplitter, with -Ptests.asserts=true and false.
• All of :lucene:misc:test with A (assertions on): 317 tests, 9 skipped, 0 failures.
• :lucene:misc:checkGoogleJavaFormat, ecjLintMain, ecjLintTest, forbiddenApisMain and forbiddenApisTest pass with A and the test (without its unused import); the build's optional eclint check is not configured on the host and was skipped.
• Earlier, with the same fix shadowed in: the hunt's, the independent reproducer's and the cross-model reviewer's tests, a no-wrapper flight-recorder check (then forcing the three copied files) and upstream TestIndexSplitter pass.

Copy with destFSDir.copyFrom(fsDir, srcName, srcName, IOContext.DEFAULT) instead of Files.copy, then sync as in A. It would also answer the request deferred in LUCENE-1959 to use the Directory abstraction instead of java.io.

Trade-offs

A larger change that replaces the JDK's file copy with Lucene's stream copy; the sync is still needed, so it does not replace A. Not run.

Fix every caller at once inside the core method.

Trade-offs

Not recommended. IndexWriter never calls SegmentInfos.commit (it calls prepareCommit, IndexWriter.java:5974, syncs the files itself, :5992-5993, then calls finishCommit, :4295), so it would not be affected, but ReplicaNode would sync twice: it runs dir.sync(indexFiles) and then infos.commit(dir) (ReplicaNode.java:365, :373). Putting the sync into prepareCommit or finishCommit instead would double-sync in IndexWriter. It would also break split() as written: the SegmentInfo that split() builds (IndexSplitter.java:118-131) never gets its file set, so SegmentInfo.files() throws IllegalStateException ("files were not computed yet", SegmentInfo.java:206-212). Read, not run.

A javadoc note telling callers to sync the destination themselves.

Trade-offs

Not recommended: it leaves a commit point that breaks Lucene's own commit rule.

Method: code reading (non-formal), in a hunt on commit and crash recovery.

• The hunt built one file-system crash model for Lucene's commit protocol: file data is durable only after its own fsync, a new name only after a directory fsync, renames are atomic. A crash-state enumerator and a TLA+ model checked IndexWriter, NRTCachingDirectory and PersistentSnapshotDeletionPolicy workloads against it; they did not cover IndexSplitter.
• To list every code path outside IndexWriter that publishes a commit point, the finder read the callers of SegmentInfos.commit and saw that IndexSplitter.split copies files and commits without syncing them.
• It wrote a JUnit test on the stock jars: a file system that records FileChannel.force per file, plus a post-crash image built under the model's rules. The test failed on main, 10.5.2 and 9.12.3 and passed with the fix sketch. It was later cut down to the one test shown here.