DirectoryTaxonomyWriter.replaceTaxonomy is not failure-atomic: when it throws, close() commits an empty taxonomy and further use commits a broken one

DirectoryTaxonomyWriter#replaceTaxonomy(Directory) swaps a writer's taxonomy for another one, usually together with IndexWriter#addIndexes on the main index. It calls indexWriter.deleteAll() and then indexWriter.addIndexes(taxoDir), and resets the writer's ordinal counter, label cache, parent arrays and epoch only after both succeed (DirectoryTaxonomyWriter.java:872-891 on main 645d4fe14a20), with no failure handling. When addIndexes throws an exception that is not an Error (an IOException while the source is read, a source with no index, a source still locked by its writer), the internal index is emptied, pending the next commit, while the writer still describes the old taxonomy. close() then commits a taxonomy with no categories at all; continued use returns old ordinals and stores new labels at the wrong ordinals, including the root's slot 0; and facet counts over the main index, which still carries the old ordinals, come out wrong with no error. It has been there since the method was added in 4.0.0-ALPHA (2012). rollback() right after the exception keeps the last commit.

1. replaceTaxonomy calls indexWriter.deleteAll() (DirectoryTaxonomyWriter.java:874), then indexWriter.addIndexes(taxoDir) (:875), and only then refreshes its reader and resets nextID, the parent arrays, the cache and the epoch (:876-890). There is no ensureOpen() and no try/catch.
2. addIndexes is all-or-nothing by itself (IndexWriter.java:3084-3086), but the deleteAll() before it is a separate pending change. For a failure inside its try (IndexWriter.java:3114), such as a read error, addIndexes treats only an Error as tragic (IndexWriter.java:3212-3220), so after an IOException the internal writer stays open with every segment about to be dropped; a lock failure on the source is thrown earlier (:3108) and leaves it open as well.
3. The taxonomy writer stays open too. getSize() returns the stale counter (:575-578); addCategory answers from the stale cache first (:385; the default LRU cache still holds the old labels of this small taxonomy); a new label gets the ordinal nextID.getAndIncrement() (:468-469) but lands at document 0 of the emptied index, the root's place.
4. close() "commits whatever changes made to it" (:283-294): the deletion, plus whatever was added since.

Committed taxonomy: root = 0, dim = 1, dim/a = 2. replaceTaxonomy with a valid, closed taxonomy whose _0.si cannot be read throws IOException. Right after it, writer.getSize() is 3 and an NRT reader on the writer sees 0. close() commits size 0; dim/a is -1. Kept open instead: addCategory(dim/a) returns 2, addCategory(dim/c) returns 3, getSize() 4; after close() the commit has size 1, with dim/c at ordinal 0. Over a main index with documents dim/a, dim/b, dim/a, a failed replace and a later session that adds dim/x and dim/y give counts x=3 y=2 instead of a=2 b=1 x=1 y=1: the reopened taxonomy is empty, the next writer adds a root at 0 (:181-185) and gives the old documents' ordinals 2 and 3 to dim/x and dim/y.

• Triggers: an I/O error while the source is read or copied (a valid source; the IOException the method declares); a source directory with no index (IndexNotFoundException, for example a rebuilt taxonomy that was never committed); from 4.7.0, a source whose own writer is still open (LockObtainFailedException). The last two are rejected only after deleteAll() ran.
• Every fault point: failing each of the 36 directory calls of a successful replace in turn breaks the taxonomy under continued use at all 36, and under close() alone at the first 32 (the last 4 fail in the reader refresh after the copy completed, so close() commits the new taxonomy).
• Not affected: a replace that succeeds; addTaxonomy (safe under all 19 single faults); a caller that calls rollback() right after the exception, which keeps the last commit.

• Who is hit. Users of DirectoryTaxonomyWriter who call replaceTaxonomy, as its javadoc suggests, "in conjunction with IndexWriter#addIndexes(Directory...) to replace both the taxonomy and the search index content". No special configuration: the plain DirectoryTaxonomyWriter(Directory) constructor with the writer's own default configuration; the Directory is the caller's choice.
• What triggers it. An I/O error while the source is read or copied, a wrong or uncommitted source directory, or (since 4.7.0) a source whose writer is still open.
• Effect. Data corruption that is committed and survives reopen: an empty taxonomy with no root after close(), or one with labels at the wrong ordinals after continued use. Main-index documents that refer to the old ordinals then get no label or another category's label, so facet counts are wrong without any error. Repair needs a taxonomy backup or a re-index of both indexes (inferred, not executed).
• Scale. Deterministic, single-threaded, any size: the repro has 3 categories.
• How likely: rare. It needs the rarely used method and an error path. The caller does see the exception, and rollback() right after it keeps the last commit.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: committed ordinal of dim/a expected:<2> but was:<-1>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:647)
       org.apache.lucene.facet.taxonomy.directory.TestReplaceTaxonomyFailure.testAddCategoryAfterReplaceTaxonomyReadFailure(TestReplaceTaxonomyFailure.java:57)

• Contracts. TaxonomyWriter.java:52-53: addCategory "returns its ordinal. If the category was already present in the taxonomy, its existing ordinal is returned." After the failure it returns 2 for dim/a, which the next commit does not have. TaxonomyReader.java:87-92: "The root category (the category with the empty path) always has the ordinal 0"; the writer's constructor says the same (DirectoryTaxonomyWriter.java:183-185). After the failure a new label is stored at ordinal 0. As Shai Erera put it on #5949 (LUCENE-4884), the taxonomy index "has the ROOT document, at doc=0". (#12337 later added a package-private deleteAll() for reindexWithNewOrdinalData, which re-adds every category, root first, before it commits; replaceTaxonomy is unchanged.)
• A legitimate trigger. A read error on a valid, closed source raises the IOException that replaceTaxonomy declares. After IndexWriter#deleteAll() and an addIndexes that fails on a source with no index (control C2), IndexWriter's maxDoc is 0, matching its pending state; the taxonomy writer instead reports getSize() 3 while an NRT reader on it sees 0.
• No documented escape hatch. The replaceTaxonomy javadoc (:867-871) says nothing about failure, and nothing says the writer must be rolled back or discarded after one.
• History. LUCENE-4633 (#5698: the reader was not refreshed) and LUCENE-5242 (#6306: the parent arrays were not reset) fixed stale writer state after a successful replace as bugs, with tests (TestDirectoryTaxonomyWriter.java:400-404, :534-553). The failure path was never discussed (LUCENE-4108, #5180) and has no test.

The reviews by another model family (OpenAI GPT-5.6) rated this case moderate (the intended-behaviour checker, who built it in full, gave medium confidence for the close()-only part, and the credibility audit pushes back on that part as well): the method was modelled on IndexWriter#deleteAll() + addIndexes() (LUCENE-4108), and IndexWriter's own deleteAll(), failed addIndexes() and close() also commit an empty index; close() is documented to commit "whatever changes made to it"; deleteAll() "can be rolled back using rollback()" (IndexWriter.java:2732-2733) and TwoPhaseCommit#rollback() "Discards any changes that have occurred since the last commit"; a caller who calls rollback() keeps the last commit; two triggers (no index, a locked source) are inputs the method should reject; the class is @lucene.experimental.

Why it fails: the caller made one call, not two, and after a declared, non-tragic exception that call left the writer contradicting its own index, which IndexWriter does not do in the same situation (control C2: deleteAll() + a failed addIndexes leaves its maxDoc and its pending state in step); the committed result is not a valid taxonomy (no root), unlike IndexWriter's empty index; nothing documents rollback() as required; the invalid inputs are rejected only after the deletion. Verdict: a defect. What survives: the close() outcome alone resembles IndexWriter semantics.

One fix was built and validated; two alternatives were not built.

A change in DirectoryTaxonomyWriter (1 file, +24/-4; tidy leaves it unchanged). The old body of replaceTaxonomy moves to a private doReplaceTaxonomy; replaceTaxonomy adds ensureOpen() and, on any Throwable, calls rollback() (failures there are added as suppressed) and rethrows. Its javadoc gains: "If this method throws an exception, this writer is rolled back to its last commit and closed, as by rollback()."

rollback() drops its ensureOpen() and does nothing on a closed writer, like close() (DirectoryTaxonomyWriter.java:289) and IndexWriter#rollback() ("this acts like close() in closeable", IndexWriter.java:2576-2577). Without that, the fix would break today's workaround: code that calls rollback() in its catch block would get AlreadyClosedException, as a run of the probe with the plain rollback-on-failure sketch shows.

public synchronized void replaceTaxonomy(Directory taxoDir) throws IOException {
  ensureOpen();
  try {
    doReplaceTaxonomy(taxoDir); // the old body, unchanged
  } catch (Throwable t) {
    try {
      rollback(); // back to the last commit, and close
    } catch (Throwable t2) {
      t.addSuppressed(t2);
    }
    throw t;
  }
}

Trade-offs

• A failure inside replaceTaxonomy rolls the writer back and closes it, like a tragic event in IndexWriter: the caller must open a new one. That held on every tested failure path; if the rollback itself throws, that failure is added as suppressed and the writer may stay open, a corner for maintainers to review.
• Categories added but not committed before the call are dropped with the rollback.
• Code that kept using the writer after a failure now gets AlreadyClosedException instead of wrong ordinals.
• rollback() on a closed writer no longer throws; no upstream test expects it to (testRollback passes).
• It protects the taxonomy writer only: replacing the taxonomy and the main index together stays a two-writer operation for the application (DirectoryTaxonomyWriter.java:867-871, TaxonomyWriter.java:36-45).

What the checks validated

• Scratch worktree of 645d4fe14a20, gradle, OpenJDK 25: the repro test fails on the stock tree and passes with the fix, assertions on and off.
• With the fix and the test, the whole lucene/facet module passes with assertions on and off: 40 suites (39 upstream + the test), 307 tests, 0 failures; TestDirectoryTaxonomyWriter (15 tests) among them.
• :lucene:facet:check -x test (checkGoogleJavaFormat, ecjLint, forbiddenApis) passes.
• Standalone runner: the finder's 4 tests, the independent reproduction's 5 tests and 4 controls pass; the intended-behaviour check's controls C1 and C2 pass, and C3, which asserts the stock bug state, fails by design with AlreadyClosedException; the failure-point enumeration finds no failing fault point for replaceTaxonomy (all 76 cases keep the last commit or refuse further use; stock: 32 and 36 of 36 fault points fail); its 10 remaining violations of 669 cases are all in the sibling reindexWithNewOrdinalData, another method.
• Not run: nightly tests, a benchmark (the change is off the hot path).

Check that the source holds an index, that its write lock can be taken and that its segment metadata reads, and only then call deleteAll().

Trade-offs

Keeps the writer usable for bad inputs, but an I/O error later in the copy still leaves the mismatch: on stock code a fault at any of the 32 directory calls inside addIndexes breaks the commit, and a pre-check runs before them. A useful addition to the recommended fix, not a replacement. (A check that reads the source's segment metadata would happen to make the repro pass, since its read error is on _0.si.)

A javadoc sentence on replaceTaxonomy: if it throws, call rollback().

Trade-offs

Cheap, but every try-with-resources caller who misses the note still commits an empty taxonomy. Resynchronising the writer from the emptied index instead was considered and not chosen: close() would still commit the deletion.

Method: non-formal first signal, then a bounded-exhaustive failure-point enumeration (the hunt's hunt unit for failure atomicity of taxonomy replace and reindex, 2026-10-28, main a9779ef3c193).

• Lead. A planning probe on the stock jars (main, 10.5.2, 9.12.3, the 10.6 RC): after replaceTaxonomy(emptyDir) threw IndexNotFoundException the writer still reported size 4, returned old ordinals, and close() committed a taxonomy of size 1 with the new label at ordinal 0; an injected read failure on _0.si led to a committed size of 0.
• JUnit. Reduced to a public-API JUnit with three outcomes (close(), continued use, facet counts over a main index) and the read-error trigger on a valid, closed source.
• Enumeration. Each directory call of a successful replace (36) was failed in turn, under two follow-ups (close() only; continued use, then close()), plus the no-index and locked-source triggers, against the invariant "the writer's answers equal the commit, which holds the old or the new taxonomy, with the root at 0". On the stock jar every fault violated it under continued use, and all but the 4 faults after the copy (in the reader refresh) under close() only; with the fix sketch none did. The same run found addTaxonomy safe under all 19 single faults.