A failed addIndexes(Directory...) leaves the incoming field schemas in the writer, so later valid documents that use those names differently are rejected until it is reopened

IndexWriter keeps one in-memory registry of every field name it knows and that field's schema (index options, doc values type, points, vectors), built from the segments when the writer opens, and checks every new document against it. addIndexes(Directory...) walks the incoming segments and, before copying each one, calls globalFieldNumberMap.addOrGet(fi) for every incoming field; addOrGet validates and registers. When the call then fails part-way (a schema conflict on a field visited later, a conflict between two incoming indexes, an index-sort mismatch on a later directory, or an IOException such as a full disk while copying a later segment), the failure path deletes the copied files and rethrows, but nothing removes the registrations. Nothing of the import is added or committed, as the javadoc promises, and the writer stays open; but for the rest of the writer session the incoming names keep the incoming schema: getFieldNames() lists a field that no segment has, and a later document that uses the name with another schema is rejected with IllegalArgumentException. Reopening the writer, or deleteAll(), clears it. The sibling addIndexes(CodecReader...) avoids this for conflicts with the destination's schema by checking every incoming field first, without registering. Every release from 4.2.0 through 10.6.0 is affected, and so are branch_11x and main. Checking up front and registering only when the copied segments are published fixes every executed trigger.

1. The registry (FieldInfos.FieldNumbers) is built once, from the segments, when the writer opens: globalFieldNumberMap = getFieldNumberMap() (IndexWriter.java:1124), which calls addOrGet for every FieldInfo of every segment (:1267-1277).
2. addIndexes(Directory...) (:3103) reads each directory's latest commit (:3126-3141). Inside the try at :3146, for each incoming segment it checks the index sort (:3152-3159), reads the segment's FieldInfos (:3177) and calls globalFieldNumberMap.addOrGet(fi) for every field, with the comment "This will throw exceptions if any of the incoming fields has an illegal schema change" (:3178-3182). Only then does it copy the segment (copySegmentAsIs, :3183).
3. addOrGet (FieldInfos.java:415-457) checks a known name against its registered schema (:421-422) and registers an unknown name with the incoming schema (:423-454). In the example, a_new is field 0 of the incoming segment, so it is registered before z_conflict fails.
4. The failure path (IndexWriter.java:3186-3192) deletes the files of the segments already copied and rethrows. The outer catch (:3212-3221) calls tragicEvent only for an Error, so after an IllegalArgumentException or an IOException the writer stays open. Nothing removes the registrations: FieldNumbers has no removal except clear() (FieldInfos.java:694-698), which only deleteAll() calls (IndexWriter.java:2800).
5. A later document reaches FieldInfos.Builder.add (FieldInfos.java:788) -> FieldNumbers.addOrGet -> verifySameSchema (:511-548) and is rejected against the leftover schema. getFieldNames() (IndexWriter.java:2135-2137) returns the registry's names (FieldInfos.java:690-692).
6. The sibling addIndexes(CodecReader...) runs "Best effort up front validations": the non-mutating verifyFieldInfo for every incoming field before any merge (IndexWriter.java:3288-3293, FieldInfos.java:401-408). With the same inputs it leaves the registry unchanged. verifyFieldInfo does not compare two incoming readers with each other; the tested conflict between two incoming readers is caught when their FieldInfos are merged and leaves no leftover either. It is not fully immune: an IOException during its merge leaves the same leftover, because the merge registers fields as it goes.
7. Order matters: the leak needs at least one new incoming field visited before the failure. With the conflicting field first, nothing new is registered and nothing leaks.

• Who is hit. Applications that use addIndexes(Directory...), whose javadoc describes building sub-indexes in parallel and merging them with it (IndexWriter.java:3076-3079), and keep using the same writer after the call fails. Then every later document that gives one of the imported names a different schema is rejected.
• Triggers that leak (executed on main): a schema conflict with the destination on a field visited after a new one; a conflict only between two incoming indexes; an index-sort mismatch on a later directory ("cannot change index sort from null to <long: "s">"); an IOException while copying a later segment, the javadoc's own "disk full" example ("simulated disk full while copying _1.cfs").
• Default configuration. A default-constructed IndexWriterConfig (no custom codec or settings); ByteBuffersDirectory in the repro, FSDirectory in a reviewer's own test; with assertions on and off. The way in is an explicit bulk-import call that fails, followed by continued use of the writer.
• Loud, no data damage. One IllegalArgumentException per affected addDocument; documents without the leftover names are unaffected, and a document that matches the leftover schema is accepted. The index on disk is sound: an NRT reader and the next commit show nothing of the failed import (no documents, stored values or FieldInfos), a pending delete stays applied, no unreferenced files remain, and CheckIndex is clean. An application that logs and drops failed documents loses them.
• How long. Until the writer is reopened or deleteAll() is called. Other threads that use the same writer see the same rejections (one shared, synchronized registry; read from the code, not executed).
• Scale. None: one document per index is enough.
• Workaround. Close and reopen the writer after a failed addIndexes(Directory...).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected: the first document with "a_new" is accepted; actual: cannot change field "a_new" from index options=DOCS to inconsistent index options=NONE
       org.junit.Assert.fail(Assert.java:89)
       org.apache.lucene.index.TestFailedAddIndexesRegistersField.testFailedAddIndexesDoesNotDefineField(TestFailedAddIndexesRegistersField.java:46)

• Contract. addIndexes(Directory...) (IndexWriter.java:3084-3086): "This method is transactional in how Exceptions are handled: it does not commit a new segments_N file until all indexes are added. This means if an Exception occurs (for example disk full), then either no indexes will have been added or they all will have been." After the failed call, part of the incoming index still decides which documents the writer accepts, and the javadoc's own disk-full example leaks the same way.
• Schema definition. MIGRATE.md:1557-1564: "The per field data-structures are implicitly defined by the first document indexed that contains a certain field." No document with a_new was ever indexed into the destination.
• getFieldNames. "all field names as visible from this IndexWriter, across all segments of the index" (IndexWriter.java:2129-2133, @lucene.experimental); no segment has a_new.
• The sibling overload validates first on purpose. addIndexes(CodecReader...) checks every incoming field with verifyFieldInfo before it changes anything (IndexWriter.java:3288-3293); the commit that added it, 698f40ad51af (LUCENE-10216, #633), lists "verify field info schemas upfront from incoming readers" and "Keep addIndexes API transactional". The Directory overload never got it.
• Precedent. lucene/CHANGES.txt:20868-20873 (LUCENE-846): a failed addIndexes that could "corrupt the instance of IndexWriter (but, not the index itself)" was fixed as a bug. :21363-21370 (LUCENE-702) is the origin of the transactional contract ("either all indices are merged or none are"); the same change fixed IndexWriter.mergeSegments and IndexReader.commit() to "keep the instance state consistent to what's actually in the index".
• Tests. TestIndexOptions.doTestChangeIndexOptionsAddIndexesDirectory (TestIndexOptions.java:125-159) asserts only the exception message, with one field, so nothing can leak there; no upstream test adds a document to, or checks the field names of, the writer after a failed addIndexes(Directory...) (TestIndexSorting and TestDocValuesIndexing.testMixedTypesViaAddIndexes only check that a second import of the same input also fails; TestIndexWriterOnDiskFull rolls the writer back).

The devil's advocate built the strongest case for "accepted limitation"; the cross-model reviews had independently rated its core (the segments_N-only transactionality, the session registry and the experimental getFieldNames) moderate: the transactional sentence is literally about segments_N, and that part holds; the registry is a session registry that only grows by design (the deleteAll javadoc, IndexWriter.java:2735-2741, says only deleteAll clears "internal data structures" such as "doc value types"; #9597 calls field infos "(almost) append-only"); getFieldNames is experimental; a field whose documents are all deleted gives the same listing and rejection with no failure at all (the open #2835), and a rejected addDocument or a failed updateDocValues leaves similar state. In #13369 (the fix of #13353), for failed documents, benwtrent tried "Delaying on setting the field info until document writes are successful" (it "causes weirdness with global field numbers") and reverting the field info when all the docs for a field fail to index ("too sharp to mess with"), and settled on another fix.

Why it fails: no javadoc, test or discussion says that a failed operation may define a field's schema. In the deleted-field and rejected-document cases a document with the field reached the indexing chain; here none ever did. #13369 is about the per-document indexing chain, where delaying caused "weirdness with global field numbers"; here the copied segments keep their own field numbers (copySegmentAsIs copies the files unchanged, IndexWriter.java:3648-3705, and the result of addOrGet is unused, :3181), and the method has a documented all-or-nothing promise that its sibling already keeps by validating first. The trigger is legal public API with the default config, and the writer stays open with no tragic exception, so continuing to use it is not misuse. Our reading: a defect (the devil's advocate stayed at "uncertain"). Residual risk: a maintainer may treat it as one design question with the other entry points ("the registry is never rolled back after a failed operation").

Two fixes were built and validated with gradle; the recommended one covers every executed trigger, the smaller one only schema conflicts. A rollback alternative was not built.

Read every incoming segment's FieldInfos before copying anything and check them against the registry and against each other (a new non-mutating FieldNumbers.verifyFieldInfos); drop the addOrGet calls from the copy loop; and register all incoming fields with one synchronized FieldNumbers.addAll (verify all, then register all) inside synchronized (this), just before segmentInfos.addAll(infos). If that last call throws, the reserved document count is released and the copied files are deleted, as for any other failure there. +53/-6 in IndexWriter.java and FieldInfos.java. The core of it:

  // before the copy loop: check, register nothing
  List<FieldInfos> incomingFieldInfos = new ArrayList<>();
  for (SegmentInfos sis : commits) {
    for (SegmentCommitInfo info : sis) {
      incomingFieldInfos.add(readFieldInfos(info));
    }
  }
  globalFieldNumberMap.verifyFieldInfos(incomingFieldInfos);

  // synchronized (this), after reserveDocs(totalMaxDoc), before segmentInfos.addAll(infos)
  try {
    globalFieldNumberMap.addAll(incomingFieldInfos); // all or nothing
  } catch (Throwable t) {
    adjustPendingNumDocs(-totalMaxDoc);
    throw t;
  }

Trade-offs

• The incoming FieldInfos are read once up front instead of inside the copy loop, and kept until publish (small).
• A concurrent document that introduces one of the incoming names with another schema while the files are being copied is now accepted, and the addIndexes call then fails at publish (all or nothing, files deleted, reservation released); today that document is rejected instead. Either way one schema wins (reasoning; concurrency not executed).
• Lock order is IndexWriter, then FieldNumbers, as elsewhere; no FieldNumbers method calls back into the writer.
• addIndexes(CodecReader...)'s IOException case (its merge registers fields as it goes) is a separate path that this change does not touch.

What the checks validated

• Gradle on 645d4fe14a20, fresh scratch worktree, seed DEADBEEF: stock, the test fails with assertions on and off. With the fix: the test plus TestAddIndexes, TestIndexOptions, TestConsistentFieldNumbers, TestFieldInfos, TestIndexSorting, TestIndexWriterOnDiskFull, TestDocValuesIndexing and TestPointValues, 9 suites, 194 tests, 0 failures; the test with assertions off passes; every lucene/core test: 749 suites, 8,898 tests, 0 failures, 322 skipped.
• checkGoogleJavaFormat passes on the final diff and the test. The suites above ran with a first version that differs only in one reflowed javadoc line; with the final diff the test plus TestAddIndexes pass again (40 tests).
• Standalone runner on the same commit with the patched classes: the test passes (-ea, -da); the independent reproducer's 5 tests go from 2 failures to 0; the devil's advocate's index-sort and disk-full probes no longer leak (getFieldNames() = [s] and [other], the a_new document accepted); the diagnostic probe shows [z_conflict] after the failed call.

The hunt's fix, +16 in IndexWriter.java only: before the copy loop, check every incoming FieldInfo with globalFieldNumberMap.verifyFieldInfo and against the other incoming FieldInfos (FieldInfo.verifySameSchema); the loop keeps its addOrGet calls.

  Map<String, FieldInfo> incomingFields = new HashMap<>();
  for (SegmentInfos sis : commits) {
    for (SegmentCommitInfo info : sis) {
      for (FieldInfo fi : readFieldInfos(info)) {
        globalFieldNumberMap.verifyFieldInfo(fi);
        FieldInfo prev = incomingFields.putIfAbsent(fi.getName(), fi);
        if (prev != null) {
          prev.verifySameSchema(fi);
        }
      }
    }
  }

Trade-offs

Smallest change and mirrors the sibling overload, but it fixes the schema-conflict triggers only: an index-sort mismatch on a later directory and an IOException while copying a later segment (the javadoc's own "disk full" example) still leak with it.

What the checks validated

• Gradle on 645d4fe14a20: the same 9 suites, 194 tests, 0 failures; checkGoogleJavaFormat passes.
• Standalone runner: the test passes; the independent reproducer's 5 tests pass; earlier reviews: the refute-lens reviewer's own test 5/5 plus 6 existing addIndexes and schema tests, and TestIndexOptions 13/13.
• The index-sort and disk-full probes still reject the a_new document with it.

Record the names this call newly registered and remove them in the catch.

Trade-offs

Needs a new removal in FieldNumbers, and another thread may start using a name between its registration and the rollback, so removal is unsafe without more bookkeeping; deferral avoids that.

Method: semi-formal (an Alloy 6.2 temporal model; the first signal was code reading while modelling, then witnessed by the tool).

• The hunt unit modelled the writer-wide field registry (FieldNumbers) against the segments' FieldInfos, with contracts for schema agreement, transactional addIndexes and no phantom names (FieldNumbersTxn.als).
• While writing the addIndexes(Directory...) action, the finder noticed that the loop registers each incoming field before copying its segment. Alloy produced a counterexample to AddIndexesTxnHolds (a failed addIndexes leaves a new name registered), while ReadersTxn, the CodecReader overload, had none in the same 6-step bound.
• A JUnit on the stock jar confirmed it, later reduced to the one-test repro. The model has no I/O failures, so it says nothing about the IOException paths; those were found by the devil's advocate's probes.