A rejected IndexWriter#updateDocValues call still registers a never-seen field, so later legal documents with that field are refused

IndexWriter#updateDocValues, softUpdateDocument(s) and tryUpdateDocValue share buildDocValuesUpdate, which, for each field of the call, first registers a never-seen name as a doc-values-only field in the writer-wide field registry and only then runs the checks that can still reject the call: the field is part of the index sort, its type is SORTED or SORTED_SET (which cannot be updated), or a later field of the same call is bad. The call never undoes the registration; only reopening the writer (or deleteAll(), which empties the index) clears it. Until then, the first legal document that indexes that field with terms, points, a skip index or another doc-values type is refused with cannot change field ... to inconsistent ..., and getFieldNames() lists a field that no document or accepted update has. The rejected call changes no document, nothing reaches disk, the writer stays open, and a reopened writer accepts the document. The default IndexWriterConfig suffices (the index-sort trigger needs the opt-in index sort). It is in every release since 7.4.0 (LUCENE-8316), in the full form since 9.0.0 (LUCENE-9334), and unchanged in 10.6.0 and on main. An all-or-nothing fix (check every field first, then register all of them in one synchronized step) passes the repro and the whole core test suite.

1. IndexWriter.buildDocValuesUpdate (IndexWriter.java:2081-2126) loops over the fields of the call. After the null and NONE checks (:2086-2093) it calls globalFieldNumberMap.verifyOrCreateDvOnlyField(f.name(), dvType, false) (:2097). For a name the writer has never seen, that creates a doc-values-only FieldInfo (index options NONE, no points, no skip index) and registers it with addOrGet(fi) (FieldInfos.java:572-594; javadoc :550-561: "If the field doesn't exist and the parameter fieldMustExist is false, we create a new field in the global field numbers").
2. Only then come the checks that can still reject the call: cannot update docvalues field involved in the index sort (:2098-2104), and case NONE: case SORTED: case SORTED_SET: default: -> can only update NUMERIC, BINARY or SORTED_NUMERIC fields (:2115-2124). In a multi-field call, any later bad field also throws after the earlier fresh fields were registered.
3. The exception leaves buildDocValuesUpdate before any update is queued (updateDocValues :2072; softUpdateDocuments :1673-1674 and softUpdateDocument :1946-1947 build the updates before the new documents are indexed; tryUpdateDocValue :1724), so no document changes. The call does not remove the registry entry.
4. The next document with the field reaches IndexingChain.initializeFieldInfo -> FieldInfos.Builder.add (FieldInfos.java:788) -> FieldNumbers.addOrGet (:415-422) -> verifySameSchema (:511-515) and is refused (stack on main: FieldInfo.verifySameIndexOptions(FieldInfo.java:279) <- verifySameSchema(FieldInfos.java:515) <- addOrGet(:422) <- Builder.add(:788) <- IndexingChain.initializeFieldInfo(IndexingChain.java:1393)).
5. The registry is rebuilt from the segments when a writer opens, so closing and reopening the writer clears it. On the same writer only deleteAll() clears it, because it drops every document and the whole registry (IndexWriter.java:2750, :2800; executed). A document that carries only the registered doc-values type is still accepted (executed).

• SORTED trigger, default config (the repro): after the rejected SortedDocValuesField("tag") update, StringField("tag") + SortedDocValuesField("tag") gets cannot change field "tag" from index options=NONE to inconsistent index options=DOCS. The same through softUpdateDocuments (with the soft-deletes field configured).
• Index-sort trigger: index sorted on price; a rejected SortedNumericDocValuesField("price", 5) update, then LongField("price", 3) gets cannot change field "price" from points dimensionCount=0, indexDimensionCount=0, numBytes=0 to inconsistent dimensionCount=1, indexDimensionCount=1, numBytes=8, while a document with only SortedNumericDocValuesField("price"), the type of the rejected update, is accepted.
• Index sort on rank (LONG): a rejected BINARY update of rank makes every later document that carries NumericDocValuesField("rank"), the type the sort requires, fail with from doc values type=BINARY to inconsistent doc values type=NUMERIC. Likewise a rejected NUMERIC update of the SORTED_NUMERIC sort field price makes documents with only SortedNumericDocValuesField("price") fail.
• Two-field call (fresh price, then a SORTED tag): price stays registered and LongField("price") is refused.
• tryUpdateDocValue (:1724) and softUpdateDocument: the same effect (executed by the independent fact check).

For the same index-sort constraint, IndexingChain.initializeFieldInfo validates the field's doc-values type against the sort (IndexingChain.java:1382-1385) before it registers the field (:1392-1393): a rejected addDocument of a SORTED field on a sort field leaves getFieldNames() empty, and a later LongField document is accepted (an independent check's control). The single-field update methods updateNumericDocValue (IndexWriter.java:1964), updateBinaryDocValue (:2010) and updateSortedNumericDocValue (:2048) pass fieldMustExist=true and never register anything.

• Who is exposed. Applications that keep using an IndexWriter after it rejected one of their updateDocValues, softUpdateDocument(s) or tryUpdateDocValue calls with IllegalArgumentException, when that call named a field the writer had never seen. The update javadocs list neither the index-sort nor the type restriction, so such a call is an easy mistake (MIGRATE.md and CHANGES state only the broader rule that an updated field must be doc-values-only).
• The natural path. In LUCENE-7659 (#8710, 2017) Adrien Grand suggested that applications "try the dv-only update and fallback to a regular update if it failed (which should be fine since dv updates are atomic)". For a never-seen field, the failed doc-values update now registers the field and the fallback document is refused.
• Default vs opt-in. The SORTED / SORTED_SET and multi-field triggers need only the default IndexWriterConfig, codec and directory; the index-sort trigger needs the opt-in index sort. Assertions on or off.
• What happens. Every later document that indexes the field with terms, points, a skip index or another doc-values type gets an IllegalArgumentException until the writer is closed and reopened (or deleteAll() empties the index), and getFieldNames() lists the field. With an index sort, a rejected update of the sort field that names another doc-values type than the sort uses stops every document that carries the sort field in the type the sort requires.
• What does not happen. No corruption, no data loss, no wrong search result: the rejected call changes no document, the committed FieldInfos lack the field, CheckIndex is clean, the writer stays open with no tragic event.
• Scale and likelihood. One rejected call blocks each never-seen field it names, up to and including the one that fails, until the writer is reopened, on an index of any size. Rare: it needs an update the writer must reject, for a field it has never seen, and an application that keeps indexing that field. Solr, Elasticsearch and OpenSearch do not reach the rejecting branches on their default paths (read in their source, not run).

Copy the file to lucene/core/src/test/org/apache/lucene/index/ and run the command. The test makes one updateDocValues call that is correctly rejected (SORTED doc values cannot be updated) and then adds one document that a fresh writer accepts; it fails if that document is refused. It uses no randomness of its own; the only randomized input is the test framework's default codec, and the same failure was seen with three seeds and three codecs. Pass -Ptests.asserts explicitly, because by default the build picks it from the seed. The other triggers (an index-sort field, softUpdateDocument(s), a two-field call) have the same root cause and are described with their executed messages above; they are not extra repros.

./gradlew :lucene:core:test --tests "org.apache.lucene.index.TestRejectedDocValuesUpdateRegistersField" -Ptests.seed=DEADBEEF -Ptests.asserts=true

# gradle, main 645d4fe14a20, seed DEADBEEF, -Ptests.asserts=true
TestRejectedDocValuesUpdateRegistersField > testRejectedUpdateDoesNotDefineField FAILED
    java.lang.AssertionError: expected: the first document with "tag" is accepted; actual: cannot change field "tag" from index options=NONE to inconsistent index options=DOCS
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:287DC055EC892FED]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.apache.lucene.index.TestRejectedDocValuesUpdateRegistersField.testRejectedUpdateDoesNotDefineField(TestRejectedDocValuesUpdateRegistersField.java:46)
1 test completed, 1 failed

# the same test with assertions off (gradle and the repro harness): the same message; on the 10.6.0 jars (on and off) and the 10.5.2 jars (off): the same message

# with the recommended fix, assertions on and off (the repro plus 8 closest suites):
:lucene:core:test (SUCCESS): 181 tests, 2 skipped

• The document path validates before it registers. IndexingChain.initializeFieldInfo checks the field's doc-values type against the index sort (IndexingChain.java:1382-1385) before fieldInfos.add (:1393); a rejected document of that kind leaves nothing behind (executed control). LUCENE-8316's code comment aims for "a consistent error message as if you try to do that during an indexing operation"; for this constraint the indexing operation leaves no state, the update does.
• The javadocs do not state the restrictions the trigger breaks. The updateDocValues javadoc ("Updates documents' DocValues fields to the given values ... All updates are atomically applied and flushed together", IndexWriter.java:2060-2062) states no index-sort or type restriction; neither do softUpdateDocument(s) and tryUpdateDocValue. MIGRATE.md (:1569-1574) and CHANGES document only that an updated field must be doc-values-only, which a never-seen field does not violate. The writer stays open after the IllegalArgumentException (no tragic event, executed), so it is meant to stay usable, and a call that did nothing then breaks the next legal call.
• Nothing says a rejected call may define a field. No javadoc, test, CHANGES or MIGRATE entry, commit message or upstream discussion found says so (LUCENE-7581 #8632, LUCENE-8316 #9363, LUCENE-9334 PR #11, #16565, #16581, open #16810). TestIndexSorting.testBadDVUpdate asserts only the rejection message.
• An accident of insertion order. Until 7.3.1 the check was read-only (if (!globalFieldNumberMap.contains(f.name(), dvType) ...) throw new IllegalArgumentException("can only update existing docvalues fields! ...")). LUCENE-8316 (0c3628920afd, 2018) turned it into a registering addOrGet placed before the LUCENE-7581 index-sort guard (4efbde4e7627, 2016) and the type branch; rejected calls are not discussed.
• Precedent. #7111 (LUCENE-6049, 2014): a flush failure caused by FieldInfos state that documents rejected with a non-aborting exception left behind was fixed as a bug (Michael McCandless: "this is a scary method and it has caused too many bugs over the years", about FieldInfos.addOrUpdate).
• Not the main contract: MIGRATE.md. "The per field data-structures are implicitly defined by the first document indexed that contains a certain field" (MIGRATE.md:1559) is contradicted by design for accepted updates, which create fields without any document (LUCENE-8316, TestMixedDocValuesUpdates.testUpdateNotExistingFieldDV).

The strongest case for "intended". IndexWriter documents no strong exception safety for its field registry. A rejected multi-field addDocument also leaves its earlier fresh fields registered, and that even persists across commit and reopen, because the rejected document stays as a deleted document (executed by independent checks). In the LUCENE-9334 review (apache/lucene-solr#2186) Mayya Sharipova asked on 2021-03-08 to confirm "A first doc with a field introduces FieldInfo for this field for the whole index, even if eventually this doc doesn't get indexed", and Adrien Grand replied on 2021-03-09 "These assumptions sound right to me." The javadoc's "atomically applied and flushed together" is about how accepted updates are applied, not a promise about the field registry after a rejection. Every trigger is a call that can never succeed, the error is loud, and integrators validate first (elastic/elasticsearch#157749 allows only fields outside the index sort).

Why it fails for the single-field cases. The analogue the committers accepted is a document that fails on a later field. For a field's own index-sort constraint the document path checks first and leaves nothing, and the update path does the reverse. The trigger is not a documented misuse, and the committer-recommended fallback pattern is exactly what breaks. Verdict: a low-severity bug; expect a maintainer to call the multi-field sub-case consistent with addDocument.

Options A and C were applied to 645d4fe14a20; A ran through gradle in a scratch worktree (removed afterwards) and, like B and C, through the repro harness with the patched classes shadowed. D needs no Lucene change.

buildDocValuesUpdate keeps the null, NONE, index-sort and type checks in its loop and builds the updates, but no longer registers anything there. After the loop, one new package-private FieldNumbers.verifyOrCreateDvOnlyFields(names, types) verifies every field that already exists (with fieldMustExist=true), rejects a call that names the same missing field with two types, and only then creates the missing fields, all under the FieldNumbers lock. +39/-4 lines in IndexWriter.java and FieldInfos.java.

diff --git a/lucene/core/src/java/org/apache/lucene/index/FieldInfos.java b/lucene/core/src/java/org/apache/lucene/index/FieldInfos.java
index 5bf4bbca524..dff8277da91 100644
--- a/lucene/core/src/java/org/apache/lucene/index/FieldInfos.java
+++ b/lucene/core/src/java/org/apache/lucene/index/FieldInfos.java
@@ -28,6 +28,7 @@ import java.util.Arrays;
 import java.util.Collection;
 import java.util.HashMap;
 import java.util.Iterator;
+import java.util.LinkedHashMap;
 import java.util.List;
 import java.util.Map;
 import java.util.Objects;
@@ -687,6 +688,36 @@ public class FieldInfos implements Iterable<FieldInfo> {
           isParentField);
     }
 
+    /**
+     * Calls {@link #verifyOrCreateDvOnlyField} with {@code fieldMustExist=false} for all fields of
+     * one doc values update, all or nothing: it first verifies every field that already exists (and
+     * that a field missing so far is named with one doc values type only), and only then creates
+     * the missing fields, so that a rejected update leaves the field numbers unchanged.
+     */
+    synchronized void verifyOrCreateDvOnlyFields(String[] fieldNames, DocValuesType[] dvTypes) {
+      Map<String, DocValuesType> missing = new LinkedHashMap<>();
+      for (int i = 0; i < fieldNames.length; i++) {
+        if (fieldProperties.containsKey(fieldNames[i])) {
+          verifyOrCreateDvOnlyField(fieldNames[i], dvTypes[i], true);
+        } else {
+          DocValuesType previous = missing.putIfAbsent(fieldNames[i], dvTypes[i]);
+          if (previous != null && previous != dvTypes[i]) {
+            throw new IllegalArgumentException(
+                "Can't update ["
+                    + dvTypes[i]
+                    + "] doc values; the field ["
+                    + fieldNames[i]
+                    + "] has inconsistent doc values' type of ["
+                    + previous
+                    + "].");
+          }
+        }
+      }
+      for (Map.Entry<String, DocValuesType> e : missing.entrySet()) {
+        verifyOrCreateDvOnlyField(e.getKey(), e.getValue(), false);
+      }
+    }
+
     synchronized Set<String> getFieldNames() {
       return Set.copyOf(fieldProperties.keySet());
     }
diff --git a/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java b/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
index 45b59fa46dd..1414f8d0a45 100644
--- a/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
+++ b/lucene/core/src/java/org/apache/lucene/index/IndexWriter.java
@@ -2080,6 +2080,8 @@ public class IndexWriter
 
   private DocValuesUpdate[] buildDocValuesUpdate(Term term, Field[] updates) {
     DocValuesUpdate[] dvUpdates = new DocValuesUpdate[updates.length];
+    String[] fieldNames = new String[updates.length];
+    DocValuesType[] dvTypes = new DocValuesType[updates.length];
     for (int i = 0; i < updates.length; i++) {
       final Field f = updates[i];
       final DocValuesType dvType = f.fieldType().docValuesType();
@@ -2091,10 +2093,6 @@ public class IndexWriter
         throw new IllegalArgumentException(
             "can only update NUMERIC or BINARY fields! field=" + f.name());
       }
-      // if this field doesn't exist we try to add it.
-      // if it exists and the DV type doesn't match or it is not DV only field,
-      // we will get an error.
-      globalFieldNumberMap.verifyOrCreateDvOnlyField(f.name(), dvType, false);
       if (config.getIndexSortFields().contains(f.name())) {
         throw new IllegalArgumentException(
             "cannot update docvalues field involved in the index sort, field="
@@ -2122,7 +2120,13 @@ public class IndexWriter
                   + ", type="
                   + dvType);
       }
+      fieldNames[i] = f.name();
+      dvTypes[i] = dvType;
     }
+    // Only once every field has passed the checks above: verify the fields that exist and create
+    // the missing ones as doc values only fields. This is all or nothing, so a rejected call leaves
+    // the global field schema unchanged.
+    globalFieldNumberMap.verifyOrCreateDvOnlyFields(fieldNames, dvTypes);
     return dvUpdates;
   }
 

Trade-offs

• A new package-private method in FieldNumbers. Cost O(fields in the call) per call; it copies nothing.
• Verification and creation run in one synchronized method, so no other thread can register a name in between (by reading the code; not stress-tested).
• Error precedence changes: the index-sort and type checks of every field, and the construction of its update (e.g. the (Long) cast at IndexWriter.java:2109), now run before the schema check of any existing field, so a call that breaks more than one rule, within one field or across fields, may report a different one of them (by reading; not an exhaustive list). No existing core test depends on that order.
• Open PR #16810 edits the same lines (and keeps the old order), so one of the two needs a rebase.

What the checks validated

• Gradle in a scratch worktree of main 645d4fe14a20, seed DEADBEEF: the repro fails stock and passes with A.
• The repro and 8 closest suites (TestIndexSorting, TestMixedDocValuesUpdates, TestNumericDocValuesUpdates, TestBinaryDocValuesUpdates, TestSortedNumericDocValuesUpdates, TestSoftDeletesRetentionMergePolicy, TestSoftDeletesDirectoryReaderWrapper, TestFieldInfos): 181 tests, 0 failures, 2 skipped, with assertions on and off.
• All of :lucene:core:test with A and the new test, assertions on: 749 classes, 8,898 tests, 0 failures, 322 skipped. :lucene:core:check -x test passes.
• Repro harness with A shadowed: the hunt's 4-test JUnit (index sort, SORTED, softUpdateDocuments, two-field call) 4 of 4 pass (stock: 4 of 4 fail); the independent reproducer's 10 tests pass.
• A probe of one call that names a fresh a as NUMERIC and as BINARY: stock rejects it and leaves getFieldNames() = [a]; A rejects it with the same message and leaves [].

Pass 1 takes globalFieldNumberMap.getFieldNames() and, per field, verifies existing names with fieldMustExist=true, then runs the index-sort and type checks; pass 2 is the old registration loop.

Trade-offs

• Not atomic: one call that names the same fresh field twice with conflicting types, or another thread registering the name between the passes, can still leave a partial registration (found by the cross-model reviews).
• getFieldNames() is Set.copyOf of every field name (FieldInfos.java:690-691), on every update call, including the per-document soft updates of soft-deletes users (by reading; not measured).

What the checks validated

With the patched class shadowed: the hunt's 4-test JUnit 4 of 4, the independent reproducer's 10 of 10, 105 existing doc-values-update and soft-deletes tests and TestIndexSorting 62 of 62 (main ea57360912ed); the one-test repro with assertions on and off on 645d4fe14a20.

Move the verifyOrCreateDvOnlyField(f.name(), dvType, false) call (IndexWriter.java:2094-2097) below the switch: 4 lines moved, no new method.

Trade-offs

Fixes the single-field triggers only: in a multi-field call an earlier fresh field is still registered before a later field fails.

What the checks validated

With the patched class shadowed on 645d4fe14a20: the one-test repro passes; the hunt's 4-test JUnit gives 3 of 4, and the two-field call still fails. The one-test repro passes with A and with C alike, so it does not tell them apart.

Do not send doc-values updates for index-sort fields or SORTED / SORTED_SET fields (check the field type and IndexWriterConfig.getIndexSortFields() before the call), or close and reopen the writer after such an IllegalArgumentException.

Trade-offs

No Lucene change; every application has to know the rules the javadocs do not state.

• Method (semi-formal): Alloy 6.2 model checking of the argument checks of the doc-values update entry points against the writer-wide field schema, in a hunt unit on SORTED_NUMERIC doc-values updates. Properties: exception safety ("a rejected call leaves the schema unchanged") and "a document is rejected only on a real conflict".
• Result. The finder spotted the order while transcribing buildDocValuesUpdate into the model; the solver then found 2-step counterexamples to both properties (a rejected two-field call that names a fresh index-sort field registers it; a later document that indexes the field with another, non-doc-values-only schema is then rejected). The two-pass fixed model has no counterexample up to 8 steps, and a reachability run shows that rejected updates and documents still occur in it (not vacuous). A first fixed variant, validating field by field inside the loop, was itself refuted by the solver for two-field calls.
• Reduction. The finder turned the trace into a public-API JUnit on the stock jars; it was later minimized to the one test above.