addIndexes(CodecReader) with a term-vector field that has zero terms writes a chunk the default codec cannot read back, and every later merge that re-encodes the segment closes the IndexWriter

IndexWriter.addIndexes(CodecReader...) copies the term vectors of a reader that is not a same-format segment through the codec-agnostic copier TermVectorsWriter.addAllDocVectors. If the reader removes terms (a FilterLeafReader that drops a term from the postings and, consistently, from the term vectors), a document whose term-vector field held only removed terms reaches the codec as a field with numTerms = 0. The default Lucene90 term-vector writer then writes one 4-byte AvgCharsPerTerm float per field of the chunk because the field has the OFFSETS flag, while its reader reads those floats only when the chunk has at least one offset, so the chunk's other documents with term vectors (a chunk holds up to 128 documents) cannot be read. The same reader lists the zero-term field but returns null from terms(), so the next merge that re-encodes the segment (a deletion rules out the bulk copy once the merge policy picks the segment) declares a field it never writes and, on 9.0.0 and later, closes the IndexWriter, again for every new writer. CheckIndex at its default level reports the index clean. Normal indexing never produces a zero-term term-vector field; the trigger is an application-written term-removing reader. Confirmed on 11 versions from 4.2.0 (2013) to 10.5.2 and main. Before 9.0.0 (merges) and 10.4.0 (chunk reads), runs without assertions could fail silently: a merge wrote one document's term vectors under another, and chunk reads returned wrong bytes or threw ArrayIndexOutOfBoundsException.

• Term vectors: an optional per-document copy of a field's terms (with positions, offsets, payloads), stored by the default codec in compressed chunks of up to 128 documents, flushed once their term bytes reach 4 KB (Lucene90TermVectorsFormat.java:68, :170).
• The copier: TermVectorsWriter.addAllDocVectors, the codec-agnostic path that writes one document's term vectors from any Fields. addIndexes(CodecReader...) through SlowCodecReaderWrapper uses it, and so does any merge of a segment with deletions (Lucene90CompressingTermVectorsWriter.java:915-931, bulk-merge test :953-967).
• A zero-term field: a document's term vectors list field a, but its Terms has no terms. A term-removing FilterLeafReader produces it when a document's field held only removed terms. The indexing chain never does: TermVectorsConsumerPerField.finish() skips fields with no terms (TermVectorsConsumerPerField.java:69-71).

1. The copier declares before it looks. It calls startDocument(vectors.size()) (TermVectorsWriter.java:242-251; the javadoc at :64-69 promises startField will then be called that many times), forwards an empty Terms as startField(numTerms = 0) (:278-289), and skips a null Terms with continue ("FieldsEnum shouldn't lie...", :267-271).
2. Writer and reader disagree about one chunk section. The writer's flushOffsets sets hasOffsets |= fd.hasOffsets for every field instance (Lucene90CompressingTermVectorsWriter.java:629) and, if set, writes one AvgCharsPerTerm float per distinct field (:643-657). The reader computes totalOffsets from the term frequencies of OFFSETS fields, to which a zero-term field adds nothing (Lucene90CompressingTermVectorsReader.java:551-561), and reads the floats only if (totalOffsets > 0) (:587-591).
3. So the chunk is shifted. With a zero-term OFFSETS field and no other offset in the chunk, the reader starts decoding the LZ4-compressed term bytes 4 bytes per field too early. It sees a zero match offset and throws IOException("offset 0 is invalid") (LZ4.java:113-116, since #15570 in 10.4.0). The zero-term document itself reads back; the other documents of the chunk that have term vectors do not.
4. And the next merge miscounts. The same reader lists the zero-term field in iterator() and size() (Lucene90CompressingTermVectorsReader.java:879-902, :948-950) but returns null from terms() ("no term", :918-921). A merge that re-encodes the segment gives that Fields to the copier, which declares the field and skips it. The writer sizes its packed field table for the declared total (Lucene90CompressingTermVectorsWriter.java:436-450, :483-498), and DirectWriter.finish() throws Wrong number of values added (:495); with no other field in the chunk, flushFieldNums fails first (:465-466). IndexWriter treats it as tragic.

The independent reproduction's documents, default codec, main a9779ef3c193: doc 0 a = "secret" (term vectors with offsets), doc 1 body = "here public text" (no offsets). After the filtered addIndexes, the chunk's sections end at byte 21; the writer put 2 floats (fields a and body) as 8 zero bytes at [21, 29) and the compressed terms at 29; the reader starts decoding at 21. Decoding from 29 gives "herepublictext"; from 21 it throws. Without offsets and with a doc 2 that is then deleted, the commit's merge sees 2 declared fields (doc 0 a, doc 1 body) and 1 written: expected: 2, got: 1. CheckIndex: level 1 (default) clean; level 2 test: term vectors ERROR [offset 0 is invalid]; level 3 a NullPointerException on the null Terms; both report the whole segment as broken. Without offsets (the intended-behaviour check's probe), level 2 is clean and only level 3 fails.

• Who is exposed. Applications that copy, prune or rewrite an index through IndexWriter.addIndexes(CodecReader...) with their own reader that removes terms from fields that store term vectors (for example stripping a blocked vocabulary while copying an index without re-analysing the source). By reading, the same copier serves merges whose readers the application wraps (MergePolicy.OneMerge#wrapForMerge) and custom codec merges (not executed). Default codec and IndexWriterConfig, any index size.
• The pattern is used. Lucene's own TestFilterLeafReader passes a term-filtering FilterLeafReader (only terms containing 'e') to addIndexes(SlowCodecReaderWrapper.wrap(reader)); it filters postings only. A 2016 java-user thread passes a filtering reader to addIndexes to archive part of an index. No Lucene-shipped reader produces the zero-term shape from an ordinarily written segment, and no downstream engine checked does.
• What it takes. Once such a copy runs, ordinary content triggers it. The merge failure needs only the emptied field and a later merge that re-encodes the segment; a deletion rules out the bulk copy once the merge policy picks the segment; in the test, one deletion of three and a commit() with the default merge policy were enough. The unreadable chunk also needs the emptied field to store offsets and no other offset in its chunk.
• What breaks. Up to 127 other documents of the chunk (those with term vectors) have unreadable term vectors (term-vector highlighting, term-vector "more like this" and term-vector APIs fail for them). On 9.0.0 and later every merge that re-encodes the segment closes the IndexWriter, and a fresh writer fails again, so indexing stops and the segment's partial deletions cannot be reclaimed by such a merge (a segment whose documents are all deleted is still dropped without one). A merge without deletions copies the bad chunk unchanged into the merged segment.
• Silent until then. addIndexes and commit succeed; CheckIndex at its default level reports clean. With offsets, levels 2 and 3 report the whole segment broken; without offsets, level 2 is clean and level 3 fails. When it reports the segment broken, -exorcise would remove all its documents.
• Older releases, without assertions. Before 9.0.0 the merge "succeeds" and writes one document's term vectors under another document; before 10.4.0 some chunk reads return wrong bytes and others throw ArrayIndexOutOfBoundsException (on 9.12.3 the security probe counted 30,572 exceptions and 10,388 silent wrong reads). With assertions on, these releases throw AssertionError.
• What stays intact. The bug adds no damage to postings, stored fields or doc values beyond what the filter itself does (it removes the term from the postings on purpose; the postings writer drops the emptied field). One side effect: when the merge fails inside commit(), that commit's deletion is not committed.
• How likely. Rare: an expert pattern, and the searches found no report since 2013.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: expected:<[y]> but was:<[java.io.IOException: offset 0 is invalid]>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.index.TestTermVectorsZeroTermField.testOtherDocumentOfChunkReadsBack(TestTermVectorsZeroTermField.java:117)

• Lucene's own contract. startDocument: "startField(...) will be called numVectorFields times" (TermVectorsWriter.java:64-69). addAllDocVectors declares vectors.size() (:251) and skips a null Terms (:268-271).
• No numTerms >= 1 anywhere. startField: "startTerm will be called numTerms times" (:74-80). Fields, Terms, TermVectors, addIndexes(CodecReader...) (only "empty segments are dropped", IndexWriter.java:3254-3275) and SlowCodecReaderWrapper ("Wraps arbitrary readers for merging", SlowCodecReaderWrapper.java:37-40) state no such rule. The test framework's AssertingTermVectorsFormat checks that the declared counts match, not that terms exist.
• Empty Terms from filtering are anticipated. Terms.getMax(): "empty: only possible from a FilteredTermsEnum..." (Terms.java:157, :174, both in getMax()); getMin() "returns null when there are no terms" (:139-141). CheckIndex tolerates the postings "ghost field" (CheckIndex.java:2244-2249) and has the TODO "really the codec should not return a field from FieldsEnum if it has no Terms... but we do this today" (:1519-1522).
• The format expected zero-term fields. The reader's numTerms == 0 branch ("no term", Lucene90CompressingTermVectorsReader.java:918-921) dates from the format's first commit in 2013, yet the reader keeps listing the field.
• Lucene's own checks accept the input. CheckIndex.testPostings and testTermVectors at MIN_LEVEL_FOR_SLOW_CHECKS report no error on the filtered reader (they are not a complete validator of arbitrary readers); the postings path drops the same emptied field and stays clean.
• Input versus output. The filter lists the field and returns a non-null, empty Terms, which nothing forbids. After the copy, the Lucene90 reader lists the field and returns null from terms(), which conflicts with Fields.terms: "This will return null if the field does not exist" (Fields.java:38). The shape the merge copier then mishandles is Lucene's own output.
• Tests and precedent. TestParallelReaderEmptyIndex.testEmptyIndexWithVectors: "Adding this index to another index should not throw any exception" (weak precedent: ordinary indexing has already omitted the empty field there). LUCENE-1918 (#2993, 2009) fixed an addAllDocVectors crash on unusual addIndexes input as a bug. 4.1.0, and 4.2.0 with the older Lucene40 term-vector format, handle the same input.

The strongest case for "not supported" (built by the devil's-advocate checker and the cross-model reviewer): the codec SPI and Fields are @lucene.experimental; the indexing chain and MemoryIndex enforce numTerms > 0 on purpose (TermVectorsConsumerPerField.java:69-71; MemoryIndex filters numTokens > 0 in its Fields iterator and size(), MemoryIndex.java:1834, :1898; a comment there, "ensure infos.numTokens > 0 invariant; needed for correct operation of terms()", stood from 2005 until LUCENE-7091 removed it in 2016); the copier's comment "FieldsEnum shouldn't lie..." shows its author assumed a listed field has terms; and in #9450 / LUCENE-8403 maintainers declined "filtered" term vectors. So "your reader should hide a field once it has no terms left". The cross-model reviewer rated this argument moderate.

Why it fails: the restriction is written nowhere; Lucene's own Lucene90 reader produces exactly such a listing, which its own copier and CheckIndex (level 3) then cannot handle; LUCENE-8403 was about term vectors that disagree with the postings, while this filter keeps them identical; and the result is not a rejected input but a segment the default codec cannot fully read and that closes every writer whose merge re-encodes it. An IllegalArgumentException at addIndexes time would be a defensible answer; silent corruption is not. Verdict: a defect (adjudication: confirmed, high confidence).

One fix was built and validated in a scratch worktree of main a9779ef3c193; a codec-side hardening was run and is not enough alone; two more options are listed.

Workaround until fixed: a term-removing reader should hide a term-vector field whose remaining Terms is empty (leave it out of the per-document Fields iterator and size()). Not executed.

In TermVectorsWriter.addAllDocVectors (TermVectorsWriter.java:235-328), first collect the fields whose Terms is non-null and has at least one term (counting with the iterator when size() is -1, as the code already does), call startDocument with that count, then write those fields. This is what the indexing chain already guarantees (TermVectorsConsumerPerField.java:69-71) and what the postings writer does with an emptied field. It is the common copier of Lucene's shipped non-bulk term-vector merge paths (a third-party codec that overrides merge(MergeState) would need its own check). It removes both symptoms: no zero-term instance is written, and a later merge of a segment whose reader lists a null-Terms field no longer miscounts. The finder's patch, with its now-unused java.util.Iterator import removed.

final List<String> fieldNames = new ArrayList<>();
final List<Terms> fieldTerms = new ArrayList<>();
final List<Integer> fieldNumTerms = new ArrayList<>();
for (String fieldName : vectors) {
  final Terms terms = vectors.terms(fieldName);
  if (terms == null) continue;
  int numTerms = (int) terms.size();   // counted with the iterator when -1
  if (numTerms == 0) continue;
  fieldNames.add(fieldName); fieldTerms.add(terms); fieldNumTerms.add(numTerms);
}
final int numFields = fieldNames.size();
startDocument(numFields);
// then write exactly these fields

Trade-offs

• Terms are counted in the first loop instead of the second, so the number of passes is unchanged; it keeps one document's field names, Terms objects and term counts in three small lists. Not benchmarked; this is already the slow per-document path.
• Already-written segments (two-phase probe: written by stock main, merged with the fix): without offsets the merge now succeeds and CheckIndex at level 3 is clean; with offsets the chunk is already unreadable, so the merge still fails reading it (offset 0 is invalid, tragic). The fix cannot repair those bytes; rebuilding the segment from the source documents avoids it, and salvage with a tolerant reader was not investigated.

What the checks validated

• Gradle in the scratch worktree, -Ptests.seed=DEADBEEF: the new test passes 3 of 3 with assertions on and off.
• The closest suites, gradle with the fix, seed DEADBEEF: core *TermVector*, TestAddIndexes, TestFilterLeafReader, TestParallelReaderEmptyIndex, TestParallelLeafReader, TestCheckIndex and the new test, 19 classes and 151 tests, 0 failures with assertions on and off; codecs *TermVector* (SimpleText, whose merges use the same copier), 20 tests, 0 failures, 1 skipped; :lucene:core:check -x test (google-java-format, ecjLint, forbiddenApis) passes.
• Repro harness with the fix shadowed: the new test 3/0 with -ea and -da; the independent reproduction's test 3/0 with -ea and -da (registration check). Earlier, on a separate build host: 92 upstream tests (TestTermVectors, TestTermVectorsWriter, TestTermVectorsReader, TestAddIndexes, TestLucene90TermVectorsFormat) pass with -ea.

In Lucene90CompressingTermVectorsWriter.flushOffsets, hasOffsets |= fd.hasOffsets && fd.numTerms > 0; (+2/-1), so the floats are written only when the reader will read them (totalOffsets > 0).

// the reader reads AvgCharsPerTerm only if the chunk has an offset (totalOffsets > 0)
hasOffsets |= fd.hasOffsets && fd.numTerms > 0;

Trade-offs

Run with the new test (-ea and -da): doc 1 now reads back as [here, public, text], but CheckIndex (level 3) still fails on the listed zero-term field and both merge tests fail as before. It only changes what the writer emits for zero-term OFFSETS instances, which normal indexing never produces. Useful next to the recommended fix, so that no other caller of the codec API can produce the mismatched layout.

Throw IllegalArgumentException from addAllDocVectors (or the Lucene90 startField) on numTerms == 0 or on a listed field with null Terms.

Trade-offs

Defensible, but it turns a valid filtered copy into an error the application must work around, and it leaves an already-written segment with a null-Terms listing permanently unmergeable. Not run.

Skip numTerms == 0 instances in TVFields.iterator() and size().

Trade-offs

Fixes the merge miscount for existing segments too, but not the layout mismatch, and touches the read path. Not run. Separately, CheckIndex.testTermVectors could report a listed field with null Terms instead of throwing a NullPointerException (CheckIndex.java:4360-4362).

Method: SMT model checking with Z3 4.12.2 (the hunt's formal tier), by an AI tool working the hunt's unit on the compressing term-vector format, on main ea57360912ed.

• The model encodes, per field instance of one chunk, the flags (positions, offsets, payloads), the term count and the term frequencies, and for each optional section of the chunk when the writer emits it and when the reader reads it, each mapped to file:line.
• Obligation L1, "AvgCharsPerTerm presence agrees for every chunk the codec API accepts", is satisfiable for 1 to 4 field instances; the counterexample is an OFFSETS instance with numTerms == 0. L2, the same under numTerms >= 1 (what the indexing chain guarantees), and L3, the block value counts, are unsatisfiable.
• The AI tool then searched for a public-API path that delivers numTerms == 0 to the writer, found addIndexes(CodecReader) with a term-filtering reader that CheckIndex accepts, and found the second symptom, the poisoned merge, by running it.
• Independent checks followed: a reproducer with its own filter and tests, a cross-model review, a devil's-advocate checker, an adjudication, a history sweep and a security probe (see the checks below).