Since 10.5.0, IndexWriter accepts a document that omits part of a field built from several field types

Lucene requires every document that has a field to give that field the same data structures (LUCENE-9334, 9.0). A documented, ordinary way to build a field is from several Field instances under one name, for example a StringField plus a SortedDocValuesField, as StringField's javadoc suggests. Since 10.5.0, a later document that carries only one part is accepted when that part's type was the first value of the field in the most recently validated document of that field in the same in-memory segment; 9.1.0-10.4.0 reject it. The frozen-FieldType cache added by #15886 caches the first value's type after the document was validated and then skips validation for later documents whose first value has that type. Whether Lucene enforces its rule now depends on value order, frozen vs unfrozen types and flushes. The same fast path keeps leftover per-document schema state, so after a rejected addBatch the next valid multi-type document is rejected once. The index stays valid; queries and sorts that use the missing structure can silently miss the document.

• Field schema / FieldInfo: the data structures a field has in a segment, fixed by the first document that contains it (MIGRATE.md, "Require consistency between data-structures on a per-field basis").
• Multi-type field: one field name filled by several Field instances of different types in a document, for example StringField (postings) + SortedDocValuesField (doc values). The field's schema is the union of the types.
• Frozen type: a FieldType after freeze(). Stock classes share static frozen types, for example StringField.TYPE_NOT_STORED, TextField.TYPE_*, StoredField.TYPE, the *DocValuesField.TYPE constants and KeywordField's types; an application can also reuse its own frozen FieldType (for example one from KnnFloatVectorField.createFieldType). LongPoint, IntPoint and the default vector-field constructors build a new type per instance.

1. For each document, processDocument builds a per-document schema for every field from all its values and initAndValidateFields compares it with the FieldInfo (:611-640, :683-697, assertSameSchema :2290-2304). The FieldSchema javadoc (:2154-2161): "If there is inconsistency, we raise an error."
2. Since 10.5.0 (#15886), PerField.reset runs for the field's first value and records its type. If it is the cached validatedFrozenFieldType, it only updates the doc id (resetJustDocId, :1821-1831, :2272-2274), the field is not validated (:622-624) and the value is not added to the schema (:633-635).
3. A later value forces validation only if its type differs from the cached one (multiValueForcesDeoptimize, :625-630, :1833-1835), and it is merged into whatever schema state the previous operation left.
4. After validation, trySetValidatedFrozenFieldType caches the first value's type if it is frozen (:695, :1837-1843), although the FieldInfo was checked against the union of all values.

• doc 0 = [StringField, SortedDocValuesField]: no cache yet, so both values enter the schema; FieldInfo = postings DOCS + doc values SORTED; StringField.TYPE_NOT_STORED is cached.
• doc 1 = [StringField]: first value has the cached type, so no schema update and no validation. Indexed with postings and no doc value.
• doc 1 = [StringField, StoredField]: the StoredField deoptimises, but the schema still holds doc 0's DOCS + SORTED, and a stored value adds nothing to it, so the check passes.
• With doc 0 = [SortedDocValuesField, StringField] the cached type is SortedDocValuesField.TYPE: a StringField-only document is rejected, a SortedDocValuesField-only one is accepted.

processBatch resets the field's schema and fills it from the batch columns (:756-783) but never clears the cache; a rejected batch throws before the cache is updated (:693, :695). After addDocument([StringField("id")]), an addBatch with a NumericDocValuesField.TYPE column for id is correctly rejected, which leaves the batch's state (index options NONE, doc values NUMERIC) in the schema. The next document [StringField, StoredField] takes the fast path for its StringField, deoptimises on the StoredField, and is compared using the batch's leftovers: index options: expected 'DOCS', but it has 'NONE'. One false rejection per rejected batch; the next document is validated from scratch.

• Who is hit. Default IndexWriterConfig and codec, assertions on or off. The cached first value must be a shared frozen type (for example StringField, TextField, StoredField, the *DocValuesField constants, KeywordField, or a frozen FieldType the application reuses), so "point first" does not trigger it. Examples: StringField + SortedDocValuesField (StringField's javadoc: "If you also need to sort on this field, separately add a SortedDocValuesField"), NumericDocValuesField + LongPoint (the numeric sort optimisation), SortedNumericDocValuesField + LongPoint (the IndexOrDocValuesQuery pattern), StoredField + doc values.
• What it takes. A later document in the same in-memory segment that omits one of the field's structures. That document is the application's own mistake; Lucene documents that it rejects it, and every tested release from 9.1.0 to 10.4.0 does.
• Sort consequence (executed). One segment of 20,000 documents, each with StringField("k", "x"): 19,999 with NumericDocValuesField + LongPoint on ts (doc-values value first; document 1 holds 999,999) and, as document 15,000, one with only NumericDocValuesField("ts", 1000000), the highest value. It is accepted; a default top-1 descending sort of the term query k:x (search(query, 1, sort)) returns 999,999, while the same search without skipping returns 1,000,000. Default codec, plain main(), main -ea/-da and 10.5.2. MIGRATE.md: the sort optimisation "relies on the assumption that the same data is stored in these points and doc values."
• Conditions of the sort consequence. Whether the document is missed depends on where it falls: at positions 10,000 to 19,999 (five tried) the default search returns 999,999, at 1,500 and 5,000 it finds it, and MatchAllDocsQuery with the same sort found it at every position tried. The verifier's JUnit shows the miss with seed 5EEDF00D; with 15 other seeds its default search did not skip at all and returned 1,000,000.
• Query consequence (executed). 5,000 documents with SortedNumericDocValuesField + LongPoint on n, then a doc-values-only document with value 7: IndexOrDocValuesQuery(n in [7, 1500]) alone does not match it (points path), but id:dv-only AND the same query does (doc-values path); the same with 6 test seeds and codecs, while 10.4.0 rejects the document. IndexOrDocValuesQuery requires its two queries to match the same documents, so this is the bypass letting the application's violation through, not a defect of that query.
• Not corruption. CheckIndex is clean before and after forceMerge(1); each structure holds exactly the documents that carried it (a probe of 350 documents: id postings 350, sorted doc values 300). On this bug's path an accepted document can lack a checked structure, but a value with a conflicting checked schema dimension (index options, norms, term vectors, doc values, points, vectors) is still rejected; stored status and attributes are not part of the check, and the sibling LFV-127 is a separate path.
• The batch face. With the experimental IndexWriter#addBatch, a schema-rejected batch makes the next valid multi-type document of that field fail once (measured: 1 of 50 per rejected batch, in each of 3 rounds), then it heals.
• Scale. Two documents are enough.
• Status. Regression in 10.5.0; still on main and branch_10x, and in 10.6.0 (released 2026-10-29).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => junit.framework.AssertionFailedError: Expected exception IllegalArgumentException but no exception was thrown
       org.apache.lucene.tests.util.LuceneTestCaseParent.expectThrows(LuceneTestCaseParent.java:1432)
       org.apache.lucene.tests.util.LuceneTestCaseParent.expectThrows(LuceneTestCaseParent.java:1418)
       org.apache.lucene.index.TestFrozenTypeCacheMultiTypeField.testDocumentWithoutDocValuesRejected(TestFrozenTypeCacheMultiTypeField.java:22)

The contract, in its own words (at ea57360912ed):

• MIGRATE.md:1557-1564: "The per field data-structures are implicitly defined by the first document indexed that contains a certain field. ... if you first index a document where a certain field is indexed with doc values and points, all subsequent documents containing this field must also have this field indexed with only doc values and points." CHANGES (9.0.0): "LUCENE-9334: Require consistency between data-structures on a per-field basis."
• IndexingChain.FieldSchema javadoc (:2154-2161): "If there is inconsistency, we raise an error. This ensures that a field has the same data structures across all documents."
• TestPerFieldConsistency#testDocWithMissingSchemaOptionsThrowsError: "the same segment: indexing a doc with a missing field throws error", for each of four structures.

Why upstream's test misses it. It drops the fields first to last. Its first rejected document drops field 0, the frozen postings type, so that document's first value has another type, which clears the cache, and the later documents are fully checked. Reversed, the first rejected document still starts with the cached type and is accepted: the reversed copy fails on main, 10.5.0, 10.5.2 and #15886's own commit, and passes on 10.4.0, 9.12.3, #15886's parent and with the fix.

Intent of #15886 (commit message): "For the common case where a field consistently uses the same frozen FieldType instance, this work is redundant — a frozen type is immutable, so its schema contribution is identical every time." "If a different type is encountered, the cache is invalidated and the full validation path runs." "A deoptimize path handles multi-valued fields where a later value uses a different type than earlier values within the same document." CHANGES lists it under 10.5.0 Optimizations. The PR had one approval and no discussion of fields made of complementary types; its tests and those of #16171 cover multi-valued fields whose types share one schema or conflict inside one document. The author's first PR comment names the motivation: light-indexing workloads where schema validation dominates processDocument; no difference in the luceneutil benchmarks, and a 10% documents-per-second gain in doc-values-oriented macro runs of Elasticsearch, together with another change.

Old behaviour. 9.1.0, 9.12.3, 10.0.0, 10.3.2, 10.4.0 and #15886's parent reject every subset document; #15886 and every later release accept them.

Context. In LUCENE-10125 (2021) a committer wrote that "the schema definition of a field to be splitted across multiple Field instances makes it challenging to improve" the LUCENE-9334 check: the case the cache gets wrong.

The cache is a deliberate hot-path optimisation keyed by object identity (the PerField comment says it lets IndexingChain "skip schema validation if fields keep using the same frozen field type"); the subset document breaks the documented rule anyway (garbage in); the LUCENE-9334 issue called the check "best-effort"; addBatch is @lucene.experimental.

Why it fails: Lucene documents that it rejects such input, upstream tests assert it, and 9.1.0-10.4.0 did so deterministically. The "best-effort" list in the LUCENE-9334 issue (#10374) names this very case: "it is illegal for a document to only enable points and another document to only enable doc values". "Best-effort" does not cover an outcome that flips with value order, frozen vs unfrozen types and flush timing. #15886 presented the change as removing redundant work, not as relaxing the rule. The comment's premise (the field keeps using one frozen type) is false for a field built from several types. And an experimental API does not license a rejected call making a later valid addDocument fail.

What a maintainer could still raise (the cross-model fact check rated the intended case moderate for the subset document and weak for the other two faces): detection may have been meant as a defence rather than a guarantee, and the triggering document is the application's own mistake; a document that omits the field entirely is always legal, while this one has the field with only part of its structures; and the two faces have separate causes and fix parts.

The fix has two independent parts, one per face, confirmed by running each part alone: the "mixed" flag alone fixes the row cases but not the batch face; clearing the cache in processBatch alone fixes the batch face but not the row cases. Option A was validated with gradle in a scratch worktree at ea57360912ed (removed after the runs), including the whole :lucene:core suite; option B with gradle on the closest suites; C with a one-line probe; D was not written.

In processDocument, a later value whose type differs from the first value's sets candidateMixed; trySetValidatedFrozenFieldType caches only when candidateMixed == false; on deoptimisation the schema is reset and rebuilt from the cached type (all earlier values in the document had it) before the new value is added.

} else {  // a later value of a field already seen in this document
  if (fieldType != pf.candidateFieldType) {
    pf.candidateMixed = true;  // several types: the first one alone does not describe the field
  }
  if (pf.multiValueForcesDeoptimize(fieldType)) {
    pf.schema.reset(docID);   // rebuild from the earlier values, all of the cached type
    updateDocFieldSchema(fieldName, pf.schema, pf.validatedFrozenFieldType);
    pf.validatedFrozenFieldType = null;
    fields[fieldsNeedInitOrValidate++] = pf;
  }
}
// PerField.reset: candidateMixed = false;
// trySetValidatedFrozenFieldType: cache only if candidateMixed == false

Trade-offs

• Almost no cost for valid documents of a field that always carries the same several types: on main every such document already deoptimises and is fully validated, because its second value's type differs from the cached one (IndexingChain.java:625-630, :1833-1835). On main its first value still takes the fast path; with the fix each such document does one schema.reset and one more updateDocFieldSchema call (a few assignments and comparisons). Not benchmarked.
• The documents that lose the fast path are single-type documents right after a multi-type document of the same field. When the other type adds a structure, these are the subset documents the bug lets through. When it adds none (StringField + StoredField, or two types with the same schema), the first valid single-type document after a multi-type one is validated instead of skipped, as on 10.4.0; the next one is cached again. Not benchmarked.
• The rebuild does not depend on every other path (such as processBatch) clearing the cache when it leaves a foreign schema state behind. It uses FieldSchema.reset, which does not clear the skip-index type; that is the separate bug LFV-116, with its own fix.

What the checks validated

• Gradle, scratch worktree at ea57360912ed: the repro and the reverse-order TestPerFieldConsistency test fail on stock (7 tests, 4 failed, seeds DEADBEEF and 1234ABCD) and pass with the fix (both seeds, and with -Ptests.asserts=false); checkGoogleJavaFormat clean.
• TestIndexOptions (the #15886 and #16171 cache tests), TestFieldInfos, TestColumnBatchIndexing, TestIndexWriterExceptions, TestDocValuesIndexing, TestPerFieldConsistency and the new class: 135 tests, 0 failures, 3 skipped. The whole :lucene:core:test (seed DEADBEEF): 748 classes, 8,870 tests, 0 failures, 322 skipped.
• Earlier in the hunt: the 6-test repro and the reversed upstream test pass; in the bounded-exhaustive harness this fix alone drops the 352 accepted-invalid sequences (of 246,848: 928 first documents x 133 single-field second documents, with and without a flush) and the 292 valid-rejected ones (of 123,424 document/batch/document triples) to 0; mismatch classes of the sibling bugs LFV-116 and LFV-127 remain until their fixes are added (all three together: 0). An Alloy check of the fixed cache found no counterexample in a small universe without addBatch; the larger checks timed out.

Instead of the rebuild, add pf.validatedFrozenFieldType = null; next to pf.schema.reset(baseDocID) in processBatch (IndexingChain.java:761); +13/-6 lines.

// processBatch, first column for this field name in the batch
pf.columnFeatures = (byte) columnFeatures;
pf.validatedFrozenFieldType = null;   // the batch replaces the field's per-document schema
pf.schema.reset(baseDocID);

Trade-offs

Simpler, and it sits on the line where the sibling bug LFV-127 (a rejected document's type promoted by addBatch) also needs pf.candidateFieldType = null;, so both batch issues can be fixed in one place. It relies on every path that leaves a foreign schema state also clearing the cache.

What the checks validated

The repro 4/4 and the hunt's repro 6/6 pass; all six probe cases are rejected (fact check A); checkGoogleJavaFormat clean; the seven closest classes: 135 tests, 0 failures, 3 skipped. The full core suite was not run with this variant.

Never cache a frozen type; every document is fully validated, as in 10.4.0.

Trade-offs

Fixes both faces by construction but gives up the validation skip #15886 added for documents that repeat one frozen type. Its author reported no difference in the luceneutil benchmarks and about 10% more documents per second in doc-values-oriented macro runs of a downstream engine, measured together with another change; nothing was benchmarked here.

What the checks validated

A one-line probe that never caches a frozen type makes an independent verifier's row, batch and sort-impact tests and the hunt's repro pass.

For example, remember the set of types of the last validated document and take the fast path only for the same set.

Trade-offs

More general and keeps a fast path for multi-type fields, but more code and per-document bookkeeping. Not written or tested.

Method: Alloy 6.2 model plus bounded-exhaustive differential testing (semi-formal: the first signal came from code reading while modelling).

• The hunting unit modelled the per-segment FieldSchema / PerField state machine of IndexingChain, including the frozen-type cache, addBatch and flushes, with a file:line mapping to the Java code.
• While writing the cache part, the AI tool saw that the cached type is the first value's type while validation runs on the union. The check RowOnlyStockCacheSound against a stateless reference of the per-document contract was satisfiable with 3-document counterexamples; with the fix, CacheFixLocalCacheSound had no counterexample in a small bounded universe without addBatch (row-only traces; the larger fixed-model checks timed out).
• A differential harness on the real jar ran every pair of a first document (928 kinds) and a second document that has only field f (133 kinds), with and without a flush in between (246,848 sequences: 352 accepted although invalid), and every document / addBatch / single-field document triple (928 x 133 = 123,424: 292 valid documents rejected); this fix removes both classes, and neither of the two sibling fixes does.
• The counterexample was minimised to a public-API JUnit test on the stock main jar (4 of 6 fail, -ea and -da).