An addBatch that declares a new term-vector field but indexes no row leaves a commit that cannot be opened

IndexWriter#addBatch (the experimental column API, 10.5.0 and later) registers the field of every column in the segment's FieldInfos before it indexes any row. For a field with term vectors that only sets a flag; the term-vectors writer is created when the first document finishes, and flush writes term-vector files only if that writer exists. A batch that indexes no row (numDocs() == 0, a later column rejected by the schema check, or a cursor that throws on its first nextDoc()) therefore leaves FieldInfos that declare term vectors and no .tvd/.tvx/.tvm files. A document already in the same in-memory segment is flushed into that segment, and the commit cannot be opened: DirectoryReader.open throws CorruptIndexException, or, with the default config and the two failure triggers, commit() and close() throw. The per-document path is safe because it always finishes the document it started.

• FieldInfos: the per-segment list of fields and what each one stores (postings, norms, term vectors, doc values, points, vectors). A reader opens one file set per feature that FieldInfos declare.
• Term vectors: an optional per-document copy of a field's terms (FieldType#setStoreTermVectors), stored in .tvd, .tvx and .tvm files.
• addBatch / ColumnBatch: the experimental column-oriented indexing API (10.5.0). A batch has numDocs() documents and a list of columns, one per field feature; each column hands out a cursor over (doc, value) pairs.
• In-memory segment: the documents an IndexWriter thread has buffered since its last flush; a flush or commit writes them as one segment.

1. IndexingChain.processBatch (IndexingChain.java:717) first loops over the columns (:726-780), then calls initAndValidateFields (:782-785). In column order, that creates the FieldInfo of each new field (:692-693) or checks an existing field's schema (assertSameSchema, :695, which throws IllegalArgumentException on a conflict).
2. Creating an indexed field calls PerField.setInvertState() (:1415-1416). For a field with term vectors it calls termVectorsWriter.setHasVectors() (:1916-1918), which only sets a flag (TermVectorsConsumer.java:112-114).
3. The term-vectors writer is created only in TermVectorsConsumer.finishDocument (:117-128), when a document finishes. flush writes term vectors only if (writer != null) (:78-89).
4. The batch path finishes documents only in its row loop (IndexingChain.java:889, termsHash.finishDocument at :937), and asks each row column for its first document before that loop (:875). No document is finished when (a) numDocs() is 0, (b) step 1 threw for a later column, after body was created, or (c) the cursor throws at :875.
5. For (a) reserveDocs returns at once (DocumentsWriterPerThread.java:226-230); for (b) and (c) updateBatch treats the exception as non-aborting and marks the batch's reserved documents deleted (:349-354). The writer keeps going, and the earlier document is flushed into the same segment.
6. SegmentCoreReaders opens the term-vectors reader whenever FieldInfos declare term vectors (SegmentCoreReaders.java:120-128) and turns the missing file into CorruptIndexException (:141-144).

The per-document path is safe: processDocument calls termsHash.startDocument() (IndexingChain.java:598) before it creates any FieldInfo and termsHash.finishDocument in its finally block (:662-681), so even a rejected document creates the writer, and flush fills empty entries.

Default codec, NoMergePolicy, one document (id = "0"), then the batch. Right after the batch the in-memory state is body(tv=true), hasVectors=true, writer=null. The committed segment _0 has maxDoc 1 (empty batch), 2 (rejected batch, 1 deleted) or 4 (cursor fails, 3 deleted), 1 live document, and 12 or 16 compound sub-files: postings, norms (also for body), stored fields, FieldInfos and, for (b), doc values, but no term vectors. DirectoryReader.open throws. The same rejection through addDocument creates the writer, writes .tvd, .tvm and .tvx, and opens with 1 document.

• Who. Applications on IndexWriter#addBatch (@lucene.experimental, IndexWriter.java:1535) or updateDocuments(Term|Query, ColumnBatch), which reaches the same code (IndexWriter.java:1554, :1570; by code reading, not executed). Released in 10.5.0, 10.5.1, 10.5.2 and, by reading, 10.6.0 (branch_10_6 has the same code).
• What it needs. A batch that declares a term-vector field new to the in-memory segment and indexes no row: empty, rejected by a later column's schema check, or a cursor that fails on its first value. At least one other live document in that segment. For a rejected batch, the term-vector column must be checked before the failing one. Default IndexWriterConfig and codec (also SimpleText, run with the empty batch only); deterministic; any scale. That another live document is needed was run for the empty batch; for the other two triggers it is read from the code. The three triggers are examples: any exception after the batch's FieldInfos are created and before its first row is finished does the same (code reading).
• What masks it. A later document indexed in the same segment before it flushes creates the writer and repairs the segment: an addDocument (run) or, by code reading, a batch with an inverted column and a row. A later doc-values-only batch does not, and an index sort does not either (run with the empty batch).
• What it does.
  • NoMergePolicy, or the default config with an empty batch: commit() succeeds and writes a commit no reader can open; with the default deletion policy it replaces the previous good commit.
  • Default config, rejected or failing-cursor batch: commit() throws CorruptIndexException (merge-on-commit opens the new segment); the writer stays open and not tragic; one more addDocument plus commit() fails again, and close() throws; on a new index nothing is ever committed. Only these two fail inside commit() because they leave the segment 50% and 75% deleted, so the default merge policy merges it at commit, which opens it (code reading). After an earlier good commit, rollback() closes the writer; a new writer opens the previous commit, without the document added since.
  • NRT: DirectoryReader.open(writer) throws the same exception.
  • CheckIndex reports the segment broken (numBadSegments=1, totLoseDocCount=1); -exorcise would drop it with its documents.
• Loud, not silent. No wrong results; every failure is an exception (CorruptIndexException, or IndexNotFoundException when a new index never got a commit). Postings, stored fields and doc values of the documents are written; only the term-vector files are missing.
• How likely. Rare in steady state: a pipeline whose batches always carry the term-vector column with rows heals itself. It sits on the error path, where one bad batch becomes an unopenable commit instead of one rejected batch.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.apache.lucene.index.CorruptIndexException: Problem reading index from ByteBuffersDirectory@f9f34a6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1437f163 (resource=ByteBuffersDirectory@f9f34a6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1437f163)
       org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:142)
       org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:95)
       org.apache.lucene.index.StandardDirectoryReader.createOrReuseSegmentReader(StandardDirectoryReader.java:296)
       org.apache.lucene.index.StandardDirectoryReader.createSegmentReaders(StandardDirectoryReader.java:239)
       [...]
     Caused by: java.io.FileNotFoundException: No sub-file with id .tvd found in compound file "_0.cfs" (fileName=_0.tvd files: [.nvd, .fnm, .nvm, .fdx, .fdt, .fdm])
       org.apache.lucene.codecs.lucene90.Lucene90CompoundReader.openInput(Lucene90CompoundReader.java:184)
       org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingTermVectorsReader.<init>(Lucene90CompressingTermVectorsReader.java:152)
       org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingTermVectorsFormat.vectorsReader(Lucene90CompressingTermVectorsFormat.java:96)
       org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:125)
       [...]

• Commit contract. IndexWriter#commit: "Commits all pending changes ... such that a reader will see the changes" (IndexWriter.java:4189-4190).
• Empty batches are legal. ColumnBatch#numDocs only requires doc ids "in the range [0, numDocs())" (ColumnBatch.java:28-31); reserveDocs returns at once for n == 0 (DocumentsWriterPerThread.java:228-230); upstream TestColumnBatchIndexing#testEmptyBatch (:924) adds an empty batch and checks that the "Writer still functions for follow-up work" (its batch has no columns).
• Batch failures are non-aborting by design. updateBatch: "the iterator threw an exception that is not aborting / go and mark all docs from this block as deleted" (DocumentsWriterPerThread.java:350-353); #16270 aligned it with updateDocuments; upstream test*ThrowMidBatchRecovers (TestColumnBatchIndexing.java:682, :740, :794, :841) throw from cursors and assert the writer recovers. addDocument's javadoc: after an exception "the index will be consistent" (IndexWriter.java:1465).
• The rule in the same code. setInvertState: "Even if no documents actually succeed in setting a norm, we still write norms for this segment" (IndexingChain.java:1912-1913). The broken segment does have norms for body; only term vectors lack the guarantee, and only on the batch path.
• Precedent. #13369 (9.11.0, closing #13353): "Fix NRT opening failure when soft deletes are enabled and the document fails to index before a point field is written", the same FieldInfo-without-file state for points, fixed as a bug.
• Untested corner. No upstream test combines term vectors with column batches (none of the 8 test files that use ColumnBatch). The #15990 review never mentions term vectors.

The strongest case for "intended" (a cross-model reviewer rated it moderate): Lucene deliberately avoids term-vector files until a document uses them. LUCENE-2776 (2010) stopped creating the term-vector writer at flush when no document had used it, and at first let the reader tolerate exactly this state: "If all documents flushed in a segment had hit non-aborting exceptions, it's possible that FieldInfos.hasVectors returns true yet the term vector files don't exist." The API is @lucene.experimental, an empty batch with columns is odd, and a failing cursor is the caller's code.

Why it fails: LUCENE-2811 removed that reader tolerance 17 days later ("it should only be opened if they exist"), and today a missing term-vector file is corruption. Since then the per-document path has kept the invariant by finishing every document it starts, including rejected ones. Empty batches and non-aborting batch failures are accepted and tested upstream. @lucene.experimental allows API changes, not a commit that cannot be opened. Before LUCENE-2776, flush created the writer for exactly this case ("all documents seen in this segment hit non-aborting exceptions"); fix option A restores that guard, keyed on hasVectors. Verdict: a defect (the independent reproducer, cross-model reviews, the devil's advocate and the adjudication agree).

Option A was applied in a scratch worktree of a9779ef3c193 and run; options B and D are design alternatives that were not run; C is not sufficient on its own.

If hasVectors is set and no document created the writer, create it in flush; the existing fill(numDocs) then writes an empty entry for every document. This is what the per-document path already writes after a rejected document. With an index sort, SortingTermVectorsConsumer.flush calls super.flush and overrides initTermVectorsWriter, so the same lines create its temporary writer.

// TermVectorsConsumer.flush, before "if (writer != null) {"
if (writer == null && hasVectors) {
  // A field with term vectors was registered (e.g. by an empty or rejected column batch) but no
  // document finished afterwards: still write (empty) term vectors, FieldInfos declare them.
  initTermVectorsWriter();
}

Trade-offs

• A segment that registered a term-vector field but has no document with term vectors now gets small term-vector files with empty entries, instead of none. Format and reader are unchanged.
• flush's assert numDocs > 0 still holds: a thread state with no documents is never flushed (guards at DocumentsWriterFlushControl.java:367, :584; asserts at DocumentsWriterPerThread.java:428, :444).
• Open draft PR #16269 refactors TermVectorsConsumer and keeps the writer != null guard; a patch may need a rebase if it lands first.

What the checks validated

• Scratch worktree of a9779ef3c193 (removed afterwards), gradle, seed DEADBEEF, assertions on and off. ./gradlew :lucene:core:tidy leaves the patch unchanged.
• The repro (3/3) and the independent reproducer's 6 tests and 4 controls pass: 13/13 with assertions on and off (stock: 9 of 13 fail).
• 18 test classes, 305 tests, 0 failures, 3 skipped, with assertions on and off: TestTermVectors, TestTermVectorsReader, TestTermVectorsWriter, TestColumnBatchIndexing, the 6 classes of org.apache.lucene.document.column, TestIndexWriterExceptions, TestIndexWriterExceptions2, TestCheckIndex, TestLucene90TermVectorsFormat, TestCompressingTermVectorsFormat and this bug's 3 test classes (the repro, the independent reproducer's tests and its controls).
• Extra check (one document, an empty term-vector batch, commit; then 1 document, termVectors().get(0) == null and a clean CheckIndex; once plain and once with an index sort, which uses SortingTermVectorsConsumer): stock 2 of 2 fail, option A 2 of 2 pass with assertions on and off.
• Earlier hunt runs: the finder's, the independent reproduction's and separate tests and the 5-test security probe pass with it; shadowed over 10.5.0, 10.5.2 and the commit-built jar it passes the three scenarios.

Call initTermVectorsWriter() from setHasVectors(), or from processBatch right after initAndValidateFields when a term-vector field was registered.

Trade-offs

setInvertState() does not throw IOException today, so it needs plumbing; the batch-only variant fixes this caller and leaves the invariant implicit for the next one; term-vector outputs open at field registration instead of at flush.

Skips the empty batch before any field is registered.

Trade-offs

Fixes trigger (a) only: the rejected-batch and failing-cursor triggers stay broken. Only useful combined with A or B.

Create FieldInfos only once the row pass starts, or roll them back when the batch fails.

Trade-offs

A much larger change to processBatch, whose column pass needs the per-field state; field numbers are shared index-wide; and it would differ from the per-document path, which keeps the FieldInfo of a rejected document.

What LUCENE-2776 did in 2010.

Trade-offs

LUCENE-2811 removed that tolerance; it would hide real corruption.

Method: bounded-exhaustive translation validation (the hunt's semi-formal tier) on the real jar, a hunt unit (columnar addBatch).

• Scope A: one column, 26 column specs (long, binary, dictionary, float and byte vector and token-stream columns; stored, inverted with every index option and term vectors with positions, offsets and payloads, every doc-values type, skip index, 1-D and 2-D points), dense and every sparse pattern for numDocs 0 to 3, a document before and/or after the batch, 4 writer configurations (default, index sort, parent field, soft deletes). Each scenario was indexed through addBatch and through the row-by-row reference, and the indexes were compared; with assertions on and off.
• Result: 29,408 scenarios; 32 corrupt indexes on the batch side, exactly the term-vector specs with numDocs = 0 and one document before the batch. A mutant (a +1 on one value) was caught (256 mismatches), so the comparison is not vacuous.
• The finder minimised it by hand into a JUnit; the rejected-batch and failing-cursor triggers came from reading processBatch afterwards. An independent reproducer, cross-model reviews (OpenAI GPT-5.6) and an intended-behaviour devil's advocate checked it before the adjudication confirmed it.