A doc-values update racing IndexWriter.deleteAll() outlives its field: every later commit fails, or an unreadable field is committed

IndexWriter keeps one writer-wide field schema (globalFieldNumberMap). Every doc-values update (updateDocValues, updateNumericDocValue, updateBinaryDocValue, updateSortedNumericDocValue, softUpdateDocument(s)) first checks its field there, or registers it, and only afterwards queues the update, holding no lock from the check to the enqueue. deleteAll() drops the queued updates and then clears the schema. An update that checked its field before the clear and is queued after it applies to documents added after deleteAll(), and writing it then finds no field (or one of another doc-values type): with assertions off every commit, NRT reopen and close() throws NullPointerException until rollback(), with assertions on a tragic AssertionError closes the writer, and in the type-conflict case the commit succeeds and persists a segment whose field cannot be read. It affects every release from 4.6.0 through 10.6.0 and main, with the default configuration.

1. updateDocValues (IndexWriter.java:2070-2079) calls buildDocValuesUpdate (:2081-2127), which checks or registers each field with globalFieldNumberMap.verifyOrCreateDvOnlyField (:2097) and then reads its value (:2109; the repro pauses here). Only then is the update queued: docWriter.updateDocValues(dvUpdates) (:2074). updateNumericDocValue (:1962-1979, check at :1964), updateBinaryDocValue (:2010) and updateSortedNumericDocValue (:2048) have the same shape; softUpdateDocument(s) build the update, then index the new document with it (:1664-1675, :1938-1948).
2. deleteAll() (:2750-2815) locks the per-thread writers and drops the queued deletes and updates (DocumentsWriter.lockAndAbortAll, DocumentsWriter.java:266-325, deleteQueue.clear() at :295 and :302), then aborts merges, drops the segments and calls globalFieldNumberMap.clear() (IndexWriter.java:2800). Queuing an update (DocumentsWriter.applyDeleteOrUpdate, :153) is synchronized like lockAndAbortAll, but that covers only the enqueue, not the earlier check.
3. An update that checked before the clear and is queued after it is ordered after the documents added since deleteAll(), so it applies to them.

• The new document's segment has no f, so the code takes it from the schema: "the field is not present in this segment so we clone the global field (which is guaranteed to exist)" (ReadersAndUpdates.java:922-923). constructFieldInfo returns null for an unknown field or another doc-values type (FieldInfos.java:659-688, :664, :666).
• Assertions on: assert fi != null (:926) throws; as an Error it is a tragic event and the writer closes itself (10.0.0 and later; on 9.x every commit throws).
• Assertions off: byName.put(fi.name, fi) (:928) throws NullPointerException. The update stays pending, so every later commit, NRT reopen and close() fails the same way; close() rolls back, and rollback() returns to the last commit.
• Type conflict (the new document indexed f as SORTED): the segment has f, the type check assert fi.getDocValuesType() == update.type (:920) is skipped with assertions off, and the NUMERIC update is written into the SORTED field and committed. Reading f then throws, and CheckIndex reports the segment broken. (Another doc-values type is rejected while the writer's schema has the field, FieldInfos.java:421-422; after deleteAll() the schema is empty, so the new document may legally define f as SORTED.)

Also affected: deleteDocuments(MatchAllDocsQuery), which returns deleteAll() (IndexWriter.java:1886-1890). Not affected, by code reading: tryUpdateDocValue, which is synchronized on the writer (:1721) like deleteAll's clear and only touches segments still in the writer (:1791-1829).

• Who is hit. Callers of deleteAll() or deleteDocuments(MatchAllDocsQuery) with concurrent updateDocValues, updateNumericDocValue, updateBinaryDocValue, updateSortedNumericDocValue (10.6.0 and main) or softUpdateDocument(s) on the same writer. Default IndexWriterConfig; the field may be new or already in the index (the repro updates an existing field). The missing-field failure is in the writer's own field-info code, before the codec is asked for its doc-values format (ReadersAndUpdates.java:914-932); the runs used the test framework's randomized codec.
• Common form: the writer is stuck. With assertions off, every commit, NRT reopen and close() fails until rollback(), which discards everything since the last commit, including other threads' documents. The last commit on disk stays intact (in the probe: maxDoc=1, doc 0). With assertions on, the writer closes itself.
• Worst form: a committed broken segment. When a document added inside the window gives the field another doc-values type, the commit succeeds (assertions off). Sorting or filtering on that field throws NullPointerException; other fields still read. On main and 10.5.2 the default merge policy picked the broken segment within 7 commits and the merge closed the writer; later writer sessions closed before their first commit. On 9.12.3 only forceMerge(1) hit it. Recovery: delete that segment's documents, commit and force-merge; CheckIndex is then clean.
• Scale. None: one committed and one added document.
• Likelihood: rare to contrived. The update must be between its check and its enqueue while deleteAll() completes and a document matching the update's term is added, and the field must not be re-registered before the next refresh or commit. A hook-free stock-API loop built to overlap the calls failed in 516 of 2,981 rounds on 1 CPU, 286 of 12,679 on 2 CPUs and 28 of 68,550 on 32 CPUs; a multi-thread stress over 1,024 field names failed in every run within 0.1-26 s, but with 1 or 8 field names no run failed in 3 x 30 s (masking). These rates describe loops built to overlap the calls, not a field frequency. The recorded searches found no user report.

Concurrency is the bug, so the test uses two short-lived threads; the latch (a NumericDocValuesField subclass whose first numericValue() call blocks, read by IndexWriter after the field check) only fixes the interleaving: 10 of 10 iterations with random seeds fail on main. Public API only; the test sits in org.apache.lucene.index like its neighbours, uses ByteBuffersDirectory, and compiles from 9.1.0 on (it needs the org.apache.lucene.tests packages). Like every LuceneTestCase it runs with the test framework's randomized codec (Asserting(Lucene104) in the recorded runs). For a release, run the same file against that release's lucene-core, lucene-codecs and lucene-test-framework jars.

deleter.join(10_000) lets the same test pass on a fixed writer in which deleteAll() waits for the paused update (then the test resumes the update first); with such a fix it takes about 10 s, and the test then only checks that commit() succeeds, not whether the update was applied or dropped. The type-conflict form (doc 1 indexes f as SORTED) is not part of this test. The same test is the suggested regression test; it passes with the recommended fix.

# gradle, main 5adff78bb168, seed DEADBEEF, assertions on
TestDeleteAllRacingDocValuesUpdate > testCommitAfterDeleteAllRacesDocValuesUpdate FAILED
    java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:C0DC9C1FE044052D]:0)
        at org.apache.lucene.index.ReadersAndUpdates.writeFieldUpdates(ReadersAndUpdates.java:926)
        at org.apache.lucene.index.ReaderPool.commit(ReaderPool.java:356)
        at org.apache.lucene.index.IndexWriter.writeReaderPool(IndexWriter.java:4119)
        at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3827)
        at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:4252)
        at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4214)
        at org.apache.lucene.index.TestDeleteAllRacingDocValuesUpdate.testCommitAfterDeleteAllRacesDocValuesUpdate(TestDeleteAllRacingDocValuesUpdate.java:109)

# the same, assertions off (-Ptests.asserts=false)
TestDeleteAllRacingDocValuesUpdate > testCommitAfterDeleteAllRacesDocValuesUpdate FAILED
    java.lang.NullPointerException: Cannot read field "name" because "fi" is null
        at org.apache.lucene.index.ReadersAndUpdates.writeFieldUpdates(ReadersAndUpdates.java:928)
        at org.apache.lucene.index.ReaderPool.commit(ReaderPool.java:356)
        ...

# standalone JUnit runner: 1 of 1 fails, assertions on / off, at the same two statements
# 10.6.0 ReadersAndUpdates.java:952 / :954, 10.5.2 :681 / :683, 9.12.3 :613 / :615

# after the first failure (supporting probe, not the repro)
# assertions off: commit #1-#3 and an NRT reopen throw the NPE; isOpen=true, tragic=null;
#                 close() throws the NPE and leaves the writer closed; last commit: maxDoc=1 (doc 0)
# assertions on:  commit #1 AssertionError, then AlreadyClosedException; isOpen=false, tragic=AssertionError

• Thread safety. Class javadoc: "IndexWriter instances are completely thread safe, meaning multiple threads can call any of its methods, concurrently" (IndexWriter.java:167-168). The deleteAll javadoc says it is "equivalent to creating a new IndexWriter with OpenMode#CREATE" (:2739-2740); its only concurrency note is that threads running forceMerge, addIndexes or forceMergeDeletes may receive MergePolicy.MergeAbortedExceptions (:2743-2745).
• Sequence numbers. Each change returns a sequence number "which expresses the effective order in which each change was applied" (:117-118). In an independent checker's run of this race the writer reported deleteAll = 5 < addDocument = 6 < updateDocValues = 7; in that order on a fresh writer the update simply applies.
• The invariant in the code. "the field is not present in this segment so we clone the global field (which is guaranteed to exist)", then assert fi != null (ReadersAndUpdates.java:922-926). LUCENE-9300 added the comment and an equivalent assertion (assert fieldNumbers.contains(update.field, update.type)); LUCENE-9334 turned it into constructFieldInfo(...) plus assert fi != null. In the LUCENE-9300 review (lucene-solr#1394) jpountz wrote: "the field infos of the writer should be updated when the doc-value update is performed. So the FieldInfo should always exist at this point and the doc-value type should always be correct."
• Precedent. LUCENE-8785 (#9829, fixed in 8.1 and 7.7.2): deleteAll racing concurrent indexing "causes assertion errors and potentially broken field attributes" (lucene/CHANGES.txt:6400-6402). TestIndexWriterDelete.testDeleteAllNoDeadLock (TestIndexWriterDelete.java:326) runs deleteAll() while other threads add documents; no test runs it concurrently with a doc-values update.

The strongest case for "undefined, so acceptable": deleteAll's concurrent semantics were never defined. Open #6070 (LUCENE-5006, 2013) asks to "define the allowed concurrent behavior eg when an app calls deleteAll while other threads are indexing"; in the LUCENE-9617 review (lucene-solr#2088) mikemccand suggested a javadoc warning that concurrent operations "may or may not be deleted, i.e. the behavior of such concurrent operations are not defined / predictable"; s1monw has asked for deleteAll to be removed; and in #9738 s1monw explains that some failures are not tragic "since a user could always call rollback()".

What to concede: which of the two calls wins is not specified, and either outcome (the update dropped, or applied to the new document) would be acceptable. Why it still fails: the warning was never added, and it would only allow the update to survive or not, not a writer that cannot commit or a committed unreadable field. In the same review rmuir called a deleteAll race that leaves "segments with unaligned field numbers" "a disaster" that would "definitely result in corruption", and mikemccand answered the removal idea with "I think we should fix its bugs instead!". deleteAll was never deprecated, and rollback() discards other threads' work after a race the application did nothing wrong in. The verifiers, including an executed works-as-designed review by another model family (OpenAI GPT-5.6), rated the argument weak. Verdict: not intended, a defect.

Three options. A was built and validated on main 5adff78bb168; B and C were not built.

Each update API (updateDocValues, updateNumericDocValue, updateBinaryDocValue, updateSortedNumericDocValue) wraps its unchanged body in the read lock; softUpdateDocument(s) take it around buildDocValuesUpdate and updateDocuments; deleteAll() wraps its unchanged body in the write lock. Each update then falls entirely before a deleteAll() (and is dropped) or entirely after it (and re-registers its field or is rejected), which also matches the sequence numbers.

private final ReentrantReadWriteLock fieldSchemaLock = new ReentrantReadWriteLock();

public long updateDocValues(Term term, Field... updates) throws IOException {
  fieldSchemaLock.readLock().lock();
  try {
    return updateDocValuesWithFieldSchemaLock(term, updates); // the unchanged old body
  } finally {
    fieldSchemaLock.readLock().unlock();
  }
}
// The same wrapper for updateNumericDocValue, updateBinaryDocValue, updateSortedNumericDocValue;
// softUpdateDocument(s) hold the read lock around buildDocValuesUpdate + updateDocuments;
// deleteAll() holds fieldSchemaLock.writeLock() around its unchanged old body.

Trade-offs

• Every doc-values update and soft update now takes a shared read lock (a CAS on a shared word), which can contend when many threads update at once; soft updates are the per-document update path of soft-delete users. No benchmark was run, and the validation shows only that the tests pass.
• deleteAll() now waits for in-flight updates, including a softUpdateDocument that is indexing (and possibly flushing) its document.
• Lock order, by code reading only (not a proof): no cycle among Lucene's own code paths, because the read lock is the first lock an update takes and deleteAll() takes the write lock before fullFlushLock. Not covered: user code that re-enters the writer while an update holds the read lock. updateDocValues and the soft updates read the caller's Field objects under the lock (numericValue() and the like; the repro overrides it; the soft updates also index their documents under it), and a ReentrantReadWriteLock cannot upgrade a read lock to the write lock, so a Field whose numericValue() calls w.deleteAll() would block forever. Callbacks the writer runs on its own threads (for example a merged-segment warmer that calls an update while deleteAll() waits for its merge) were not analysed. The lock order and reentrancy still need a review, and a test with such a re-entering Field.
• Open PR #16810 rewrites the index-sort check inside the same update methods; whichever lands second needs a rebase.

What the checks validated

• Gradle in a clean scratch worktree of 5adff78bb168, OpenJDK 25.0.3, seed DEADBEEF: stock, the repro fails with assertions on and off; with the fix it passes on and off (about 10 s: the test waits to see whether deleteAll() blocks).
• The whole org.apache.lucene.index test package with the repro in it: 1,398 tests (gradle's count), 0 failures, 56 skipped, in 214 test classes, including TestIndexWriter (119), TestIndexWriterDelete (29, with the concurrent testDeleteAllNoDeadLock), TestIndexWriterExceptions (40), TestIndexWriterWithThreads, TestNumericDocValuesUpdates (34), TestBinaryDocValuesUpdates (29), TestMixedDocValuesUpdates, TestDocValuesIndexing, TestSoftDeletesRetentionMergePolicy and TestIndexingSequenceNumbers.
• :lucene:core:check -x test (google-java-format, ecj lint, forbidden APIs, licenses) passes with the fix and the test in place.
• Earlier, on main a9779ef3c193, the same locking design made the latched tests of independent checkers pass, a 1-CPU hook-free stress fail 0 of 3,284 rounds, and a 30 s multi-thread stress (13,374 deleteAll calls, 671,147 updates) fail none.

In writeFieldUpdates, when the schema lacks the field or has another type, re-register the field or skip the update.

Trade-offs

Small and adds nothing to the update path, but each outcome can contradict the sequence numbers: skipping drops an update the writer ordered after deleteAll(), re-registering accepts an updateNumericDocValue that the sequential order would reject, and a type conflict has no good answer.

applyDeleteOrUpdate and lockAndAbortAll already synchronize on DocumentsWriter; do the field check inside that critical section and clear the schema in the same one as the queue.

Trade-offs

No new lock, but deleteAll() clears the schema only after aborting merges, and a running merge registers fields in the same schema (SegmentMerger.java:75, FieldInfos.java:788), so the clear cannot simply move earlier; updates would still have to be held off until it.

Method: semi-formal (a hypothesis formed by code reading while modelling, witnessed by a model checker), on main a9779ef3c193.

• The TLA+ model covers deleteAll, document adds, flushes, addIndexes registration and publication, and a doc-values update's field check and enqueue as separate actions, with a map from each model step to the Java file:line.
• TLC 1.8.0 with deadlock checking on reported a violation of the invariant NoApplyFailure in a 6-state trace: field check, deleteAll, document add, flush, enqueue. A variant with background sorting failed the same way. The configurations with an atomic check-and-enqueue, and the one without deleteAll, reported no error.
• A latched JUnit test on the stock jars then failed with assertions on and off, on main, 10.5.2 and 9.12.3.