A deleteAll() that overlaps addIndexes erases the added fields from the writer's schema, so a later commit cannot be opened by an IndexWriter

IndexWriter.addIndexes(Directory...) registers every incoming field in the writer-wide schema (globalFieldNumberMap.addOrGet, IndexWriter.java:3177-3181 at 5adff78bb168) and copies the segment files (:3183) without holding the IndexWriter lock; only then does it publish the copied segments under synchronized (this) (:3194-3210), without registering or checking the fields again. deleteAll() clears that schema under the same lock (globalFieldNumberMap.clear(), :2800) and waits for running merges, not for an addIndexes(Directory...) call that is copying. A deleteAll() in that window therefore leaves a published segment whose fields the writer no longer knows. A later document can register one of them with another doc-values type; the writer accepts it and the commit succeeds, but the next new IndexWriter rebuilds the schema from every segment (:1274) and throws IllegalArgumentException: cannot change field "f" from doc values type=NUMERIC to inconsistent doc values type=SORTED. Only OpenMode.CREATE, which drops every document, opens a writer on it. addIndexes(CodecReader...) has the same gap between its merge and its publication (:3350-3368). Default configuration, public API, every release since 4.2.0 (2013). The fix registers the incoming fields again inside the publication block.

// IndexWriter.addIndexes(Directory...), main 5adff78bb168           IndexWriter.java:3103
flush(false, true);                                                  // :3119
for (SegmentCommitInfo info : sis) {               // no IndexWriter lock held in this loop
  FieldInfos fis = readFieldInfos(info);                             // :3177
  for (FieldInfo fi : fis) {
    // This will throw exceptions if any of the incoming fields
    // has an illegal schema change
    globalFieldNumberMap.addOrGet(fi);                               // :3181  (1) f = NUMERIC
  }
  infos.add(copySegmentAsIs(info, newSegName, context));             // :3183  file copy: the window
}
synchronized (this) {                                                // :3194
  ensureOpen();
  reserveDocs(totalMaxDoc);                                          // :3199
  seqNo = docWriter.getNextSequenceNumber();                         // :3201
  segmentInfos.addAll(infos);                                        // :3209  (3) publish, no re-check
  checkpoint();
}

// IndexWriter.deleteAll()                                           IndexWriter.java:2750
synchronized (fullFlushLock) {                                       // :2764
  try (Closeable _ = docWriter.lockAndAbortAll()) {                  // :2765  stops indexing threads
    synchronized (this) {
      abortMerges();                // :2771; waits for merges only (:2849), not for addIndexes(Directory...)
      segmentInfos.clear();                                          // :2785
      globalFieldNumberMap.clear();                                  // :2800  (2) f forgotten
      return docWriter.getNextSequenceNumber();                      // :2801

1. addIndexes registers f as NUMERIC. The code comment says this registration is its schema check (:3179-3180).
2. deleteAll runs to completion while the files are copied: it drops all segments, then clears the schema. Its stop-the-world covers indexing threads (lockAndAbortAll) and merges, including the merges of addIndexes(CodecReader...) (runningAddIndexesMerges, :2849); an addIndexes(Directory...) call that is copying is in neither.
3. addIndexes publishes the copied segment. It survives deleteAll (it is added after the clear), but the schema has no f: getFieldNames() returns [] (supplementary probe).
4. The SORTED document registers f afresh (field number 0) and is accepted. The commit holds f as NUMERIC (number 1) in the copied segment and SORTED (number 0) in the new one (probe).
5. A new IndexWriter rebuilds the schema from every segment (getFieldNumberMap, :1267-1278, addOrGet at :1274); FieldNumbers.addOrGet (FieldInfos.java:415) rejects the second type (FieldInfo.verifySameDocValuesType, FieldInfo.java:297).

addIndexes(CodecReader...) has the same shape: its merge registers the incoming fields, and the merged segment is published later in a separate synchronized (this) block (:3350-3368) without registering again. deleteAll waits for that merge (LUCENE-9309, #10349) but not for the gap between its end and the publication; a two-method test (one per overload) fails for both on main 5adff78bb168 (assertions off) and earlier on 645d4fe14a20, and passes for both with the fix (supplementary probe).

• Who is hit. An application that calls deleteAll(), or deleteDocuments(MatchAllDocsQuery) (which IndexWriter.java:1886-1889 routes to deleteAll()), on one thread while another thread's addIndexes(Directory...) is copying files, or while an addIndexes(CodecReader...) is between its merge and its publication. For the hard failure it then indexes a document that gives one of the added fields another doc-values type; deleteAll's javadoc says doc-values types "are cleared" (:2739), so an application that wipes an index to re-define its schema is the natural case.
• Default vs opt-in. Default IndexWriterConfig, codec, directory, merge policy and ConcurrentMergeScheduler; public API only.
• Scale and timing. No scale is needed. The addIndexes(Directory...) window is the whole file copy, so it grows with the source index. Without any hook, a loop built to overlap the two calls (4,000-document source, assertions off, 400 rounds each) lost the schema in 39 of 400 rounds for addIndexes(Directory...) and 4 of 400 for addIndexes(CodecReader...) (a supplementary stress run on main a9779ef3c193, not the repro). The loop starts both threads together and varies a short spin delay per round, so it shows the window is reached without a hook, not how often an application would hit it.
• Silent, then lasting. Nothing fails when the document is added or committed. Afterwards no IndexWriter opens the index with the default OpenMode (CREATE_OR_APPEND), and by reading the code APPEND takes the same path; OpenMode.CREATE opens it but drops every document. Readers still open the commit (2 documents), but f has two doc-values types across segments, so no single sort type fits every segment: a sort on f as a number throws IllegalStateException: unexpected docvalues type SORTED for field 'f' (expected=NUMERIC), and by reading a string sort would fail on the NUMERIC segment. CheckIndex at its default level reports the index clean. A merge of the two segments in the still-open writer (forceMerge(1)) is a tragic event: the writer closes itself with the same IllegalArgumentException (supplementary probe re-run on 5adff78bb168; that run reports 1 of its 5 cases failed because the test framework charged the next case with the merge thread's uncaught exception from the forceMerge case, the same tragic IllegalArgumentException; every case printed its full output).
• Without the re-typing document the effect stays inside the open writer and ends when it is reopened: getFieldNames() (@lucene.experimental) omits the added fields, and updateNumericDocValue(term, "f", ...) is rejected with "Can't update [NUMERIC] doc values; the field [f] doesn't exist." (same probe).
• Not only doc-values types. An earlier test with an indexed field whose index options change (DOCS to NONE) failed the same way at the reopen, for both overloads ("cannot change field \"f\" from index options=DOCS to inconsistent index options=NONE", main a9779ef3c193).
• Bound. Unaligned field numbers alone (new names taking the copied segment's numbers) do not corrupt data: forceMerge(1) keeps every value with its field and a new writer opens. The damage needs a schema conflict.
• How likely. Rare: wiping an index while adding another one into it, then re-typing an added field, is an unusual sequence, and no user report was found. But the window is not a few statements; it is the whole copy.

Put the file in lucene/core/src/test/org/apache/lucene/index/ and run the command below. It sits in org.apache.lucene.index like the upstream tests but uses nothing package-private. The latch fires after addIndexes has registered the source's fields and before any file of the new segment exists; no sleeps, no loops.

The test framework picks a random default codec; the failing check is the writer's schema check, not codec code, and the test failed with every seed tried (DEADBEEF here; 1, C0FFEE, 5EED1234, 5EED0491 and 5EED0492 on earlier mains). The file is the minimal repro test with two lint-only edits for ./gradlew check: one over-long comment line shortened, and the unused catch parameter named _.

Regression test: this same test; it passes with the recommended fix. With the "deleteAll waits" alternative (option B) deleteAll() must move to its own thread, since it would wait for the held addIndexes (and the await/join calls, which have no timeout, would then need one). The unnamed catch parameter _ needs Java 22 or later: fine on main (Java 25), but a backport of the test to a Java 21 branch such as branch_10_6 needs a named parameter.

TestDeleteAllDuringAddIndexes > testDeleteAllWhileAddIndexesCopiesFiles FAILED
    java.lang.IllegalArgumentException: cannot change field "f" from doc values type=NUMERIC to inconsistent doc values type=SORTED
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:85C4BDEAC9E99FE6]:0)
        at org.apache.lucene.index.FieldInfo.verifySameDocValuesType(FieldInfo.java:297)
        at org.apache.lucene.index.FieldInfos$FieldNumbers.verifySameSchema(FieldInfos.java:524)
        at org.apache.lucene.index.FieldInfos$FieldNumbers.addOrGet(FieldInfos.java:422)
        at org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:1274)
        at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1124)
        at org.apache.lucene.index.TestDeleteAllDuringAddIndexes.testDeleteAllWhileAddIndexesCopiesFiles(TestDeleteAllDuringAddIndexes.java:85)

# supplementary probe (not the repro), main 5adff78bb168, assertions off: the two orders and the overlap
PROBE before deleteAll.seqNo=2 addIndexes.seqNo=4          <- deleteAll first
PROBE before SORTED doc for f rejected: cannot change field "f" from doc values type=NUMERIC to inconsistent doc values type=SORTED
PROBE before new IndexWriter(APPEND default) ok
PROBE after deleteAll.seqNo=4 addIndexes.seqNo=2           <- addIndexes first
PROBE after NRT numDocs after both calls=0 leaves=0
PROBE after new IndexWriter(APPEND default) ok
PROBE race deleteAll.seqNo=3 addIndexes.seqNo=4            <- the overlap
PROBE race NRT numDocs after both calls=1 leaves=1
PROBE race getFieldNames()=[]
PROBE race SORTED doc for f accepted
PROBE race DirectoryReader.open(commit) ok, numDocs=2 leaves [maxDoc=1 f=NUMERIC#1][maxDoc=1 f=SORTED#0]
PROBE race CheckIndex(default level) clean=true
PROBE race new IndexWriter(APPEND default) IllegalArgumentException: cannot change field "f" from doc values type=NUMERIC to inconsistent doc values type=SORTED
PROBE race new IndexWriter(CREATE) ok

• Thread safety. Class javadoc, IndexWriter.java:167-168: "IndexWriter instances are completely thread safe, meaning multiple threads can call any of its methods, concurrently." The only "no other threads" precondition is on close() (:1359); the test closes after the second thread has finished.
• Effective order. Each change returns a sequence number "which expresses the effective order in which each change was applied" (:117-118). Here deleteAll (3) precedes addIndexes (4).
• deleteAll. It clears doc-values types, is "equivalent to creating a new IndexWriter with OpenMode#CREATE", and for other threads running addIndexes(CodecReader[]) names one consequence: "they may receive MergePolicy.MergeAbortedExceptions" (:2737-2745). Its first version (64a55ede8b5, LUCENE-1705, 2009) said the same for "any of the addIndexes methods".
• The schema rule. FieldInfos.java:358-360: "to enforce that a given field never changes DV type, even across segments / IndexWriter sessions". addIndexes relies on its registration as the check: "This will throw exceptions if any of the incoming fields has an illegal schema change" (IndexWriter.java:3179-3180).
• Tests. TestDocValuesIndexing.testMixedTypesViaAddIndexes (:592) and testIllegalTypeChangeAcrossSegments (:648) assert that a type conflict with an added or earlier segment is rejected; testTypeChangeAfterDeleteAll (:690), testTypeChangeAfterCommitAndDeleteAll (:705) and testTypeChangeAfterCloseAndDeleteAll (:671) assert that after deleteAll a field may change from NUMERIC to SORTED. The race applies deleteAll's permission to a segment that survives deleteAll. TestIndexWriterDelete.testDeleteAllNoDeadLock (:326) runs deleteAll concurrently with indexing on purpose.
• History. LUCENE-8785 (#9829, fixed in 7.7.2 and 8.2.0) fixed "potentially broken field attributes in the IndexWriter when IndexWriter#deleteAll is called while actively indexing" as a bug (lucene/CHANGES.txt:6400-6402). On java-user@ (2014-05-22, "IndexWriter.addIndexes() multithread correct?") Uwe Schindler answered that addIndexes is "thread safe, like all other methods in IndexWriter".

The argument: concurrent deleteAll has no defined semantics; it is a stop-the-world call, and users who need a clean slate should close the writer and open one with OpenMode.CREATE. In the lucene-solr#2088 review (2020) rmuir asked whether FieldNumbers.clear() is safe while another thread indexes; msfroh and s1monw answered that deleteAll stops the world (IW lock, all DWPTs locked, merges aborted), and s1monw added that they had asked to remove deleteAll and would rather have users re-create the index. mikemccand proposed a javadoc warning that concurrent operations "may or may not be deleted, i.e. the behavior of such concurrent operations are not defined / predictable". Open #6070 asks to "define the allowed concurrent behavior eg when an app calls deleteAll while other threads are indexing".

Why it fails: the warning was never added, and even as proposed it allows either outcome for the overlapping operation, not a commit that the next writer refuses to open. The stop-the-world covers indexing threads and merges; addIndexes(Directory...) registers and copies outside both, so the 2020 safety argument does not reach this path. deleteAll is public, documented and not deprecated, its javadoc expects concurrent addIndexes callers, and in the same thread mikemccand wrote "I think we should fix its bugs instead!" (2020-12-01). The re-typing document is allowed after deleteAll (the tests above); the writer's own schema check, which in the reported order should reject it, accepts it. One point stands: thread safety alone does not promise a particular outcome for a deleteAll and an addIndexes that overlap. The claim is narrower: no order, and no outcome a maintainer might call undefined but legal, should leave a commit, written by the writer itself, that the next writer refuses to open. Two executed reviews by another model family (OpenAI GPT-5.6) and an intended-behaviour verifier rated the intended-behaviour argument weak.

One fix was built and validated in a scratch worktree of 5adff78bb168; the other two are design alternatives a maintainer may prefer. Coordinate with LFV-635's fix (the same clear(), another caller) and with LFV-124's fix (a failed addIndexes(Directory...) leaves its fields registered), which edits the same publication block. LFV-124's recommended fix checks every incoming field up front without registering it and then registers them all at once, all or nothing, with one synchronized FieldNumbers.addAll inside the synchronized (this) publication block. Applied on 5adff78bb168, it also makes this bug's test pass (assertions on and off), and in the two-overload test it fixes the addIndexes(Directory...) case but not the addIndexes(CodecReader...) one. So the two fixes fit together: LFV-124's register-at-publish for addIndexes(Directory...) plus this bug's re-registration (all or nothing) in the addIndexes(CodecReader...) publication block.

In both publication blocks, right after ensureOpen() (IndexWriter.java:3196 and :3354), call a new registerSchema(infos) before reserveDocs. It reads each incoming segment's FieldInfos and calls globalFieldNumberMap.addOrGet while holding the monitor that deleteAll clears under, so registration and publication become one step relative to it. The early loop stays as a fail-fast check before the copy. If a document registered a conflicting type after the deleteAll, the call throws IllegalArgumentException inside the block and the existing handlers delete the copied files (:3202-3208, :3358-3365): the outcome of the legal order deleteAll, addDocument, addIndexes (by reading the code; not executed).

// in both publication blocks, right before reserveDocs(...):
registerSchema(infos);

private void registerSchema(List<SegmentCommitInfo> infos) throws IOException {
  assert Thread.holdsLock(this);
  for (SegmentCommitInfo info : infos) {
    for (FieldInfo fi : readFieldInfos(info)) {
      globalFieldNumberMap.addOrGet(fi);
    }
  }
}

Trade-offs

• Re-reads each incoming segment's FieldInfos while holding the IndexWriter monitor (one small file per segment). A variant that keeps the FieldInfos read in the first loop and registers those avoids the re-read; a Directory-only form of it passed TestAddIndexes (39/39) and TestIndexWriterDelete (21/21) on main a9779ef3c193, whose IndexWriter.java differs from 5adff78bb168 only in the #16963 InfoStream line.
• Keeping the early addOrGet loop keeps LFV-124's leak (a call that fails after it leaves its fields registered), and registerSchema registers field by field, so a conflict found at publication leaves the incoming fields registered before it (by reading; not executed). An all-or-nothing registration (verify every field first, then register, as LFV-124's FieldNumbers.addAll does) avoids both; with LFV-124's fix in place, only the addIndexes(CodecReader...) publication block still needs this change (the combination was not built).
• In the common case (no deleteAll) the extra addOrGet calls find the same schema and change nothing.

What the checks validated

• Gradle, clean scratch worktree of 5adff78bb168, OpenJDK 25.0.3, seed DEADBEEF: the test fails on stock with assertions on and off and passes with the fix, on and off.
• With the fix: TestAddIndexes 39/39, TestIndexWriterDelete 23 passed + 6 skipped, TestDocValuesIndexing 33/33; the gradle selection --tests org.apache.lucene.index.* (one seed, no nightly or monster runs): 215 classes, 1,398 tests, 0 failures, 56 skipped; :lucene:core:check -x test passes with the fix and the test in place.
• Both overloads: a two-method test (one per overload) fails 2 of 2 on stock and passes 2 of 2 with the same change (main 5adff78bb168, assertions off; earlier on 645d4fe14a20).

Count running addIndexes(Directory...) calls, as abortMerges already counts runningAddIndexesMerges, and have deleteAll wait for them.

Trade-offs

deleteAll then blocks for up to a whole copy while it holds the full-flush lock and every DWPT lock, which needs care to avoid a new lock-order problem (compare LFV-1137). The regression test must run deleteAll() on its own thread. Not built.

FieldNumbers.clear() bumps a counter; addIndexes records it when it registers and, under the publication lock, re-registers (as the recommended option) or fails if it changed.

Trade-offs

Avoids the re-read when no deleteAll intervened, but adds state; failing instead of re-registering would give callers a new exception for a legal call order. Not built.

Method: semi-formal (a hypothesis formed by code reading while writing a TLA+ model, witnessed by the model checker), 2026-10-26.

• A TLA+ model of the writer-wide field schema treats the field registration of addIndexes (outside the IndexWriter monitor) and its publication (inside it) as separate actions, with a map from each model step to the Java file:line. Writing that map raised the question of what deleteAll does in between.
• TLC 1.8.0 reported the invariant SchemaCovers (every field of a published segment is in the schema) violated in a 4-state trace: register, deleteAll, publish. The same model is clean with the fix modelled and without deleteAll. A re-run of the model on main 645d4fe14a20 (code identical to its map) gave the same trace.
• Two other analyses in the hunt reached the same interleaving independently the same evening: a model-based linearizability test that held addIndexes at 7 points against 7 concurrent operations (49 combinations; the only 2 outside the sequential model were deleteAll at two points inside the copy loop) and a second TLA+ model of addIndexes and its merge source.
• The three were reduced to the one public-API JUnit test of this page, with a latch so that it is deterministic.