A rejected document with a skip-indexed doc value makes later valid documents with that field fail until the next flush

IndexWriter checks that a field has the same data structures in every document. For that, IndexingChain keeps a per-field record of what the current document contains (FieldSchema), which its javadoc says is reset with every new document. Its reset() resets every member it compares except the doc-values skip index type added in 10.0. So when a document that carries a skip-indexed doc value for field f (SortedDocValuesField.indexedField and the SortedSet, Numeric and SortedNumeric variants, or FieldType#setDocValuesSkipIndexType) is rejected, the record keeps RANGE while its doc-values type goes back to NONE. Every later document that has f without doc values then fails in the same in-memory segment, with doc values skip index type: expected 'NONE', but it has 'RANGE' or field 'f' cannot have docValuesSkipIndexType=RANGE with doc values type NONE, until the next flush, NRT reopen or commit. The failure is loud: each affected addDocument throws, the writer stays open and no corruption was seen; applications that drop failed documents lose valid data. Introduced with the skip index itself (#13449), it affects every release from 10.0.0 to 10.6.0 and main; one added line fixes it.

1. The record. IndexingChain.FieldSchema (IndexingChain.java:2209-2360) holds docValuesType and docValuesSkipIndex (:2224-2225) among other members. Javadoc (:2210-2215): "With every new document this schema is reset. ... we compare the built schema of the current document with the corresponding FieldInfo ... If there is inconsistency, we raise an error."
2. The write. updateDocFieldSchema calls setDocValues(type, skip) only for a field with doc values (:1564-1565). setDocValues (:2291-2300) stores both values, but only while docValuesType == NONE (:2293-2295).
3. The reset. For each new document, PerField.reset (:1876-1886) calls schema.reset(docId) unless the field type is the cached frozen type that already validated. reset(int) (:2331-2343) resets omitNorms, storeTermVector, indexOptions, docValuesType, the three point and the three vector members, and not docValuesSkipIndex.
4. The poisoned document. Doc 1 sets (SORTED, RANGE) and is rejected by assertSameSchema (:2345-2359). Its second instance of f has a different field type, so the frozen-type fast path is dropped (:627-631) and, because validation threw, not re-armed (:697 runs only after success). So doc 2 goes through schema.reset, which leaves (NONE, RANGE). (The fast path exists since 10.5.0; before that every document goes through reset, with the same result.)
5. The two failures (initAndValidateFields, :685-699). When f has a FieldInfo in the segment, assertSameSchema compares the stale RANGE with the FieldInfo's NONE (:2350): "doc values skip index type: expected 'NONE', but it has 'RANGE'". When it has none yet, initializeFieldInfo passes (NONE, RANGE) to new FieldInfo (:1402-1403), whose constructor always runs checkConsistency() (FieldInfo.java:125) and rejects it (:160-168). Either way the next document takes the same path again.
6. Why it lasts. A rejected document is non-aborting: it keeps its doc id and is marked deleted, and deleteLastDocs does not trigger a flush (DocumentsWriterPerThread.java:283-307, :399-415). It does not even reach the flush check: DocumentsWriter.updateDocuments calls flushControl.doAfterDocument (DocumentsWriter.java:434) only after dwpt.updateDocuments (:427-428) returns. So the window ends when accepted documents trigger a RAM flush, or at an explicit flush, NRT reopen or commit (all three executed); a new in-memory segment starts with new records (DocumentsWriterPerThread.java:193). One more exit, executed in the fact check's probe: a later document that gives f doc values without a skip index is itself rejected, but setDocValues first overwrites both members, so the next StringField f document is accepted. A later skip-indexed document leaves RANGE behind again.

• It persists. After one rejected document, 20,000 of 20,000 later valid documents with f were rejected while 20,000 of 20,000 interleaved documents without f were accepted; an NRT reopen ended it, and with a 1 MB RAM buffer and other documents being indexed the first RAM-triggered flush ended it after 2,074 rejections. A second thread on the same writer, run after the first, had 100 of 100 valid documents rejected (consistent with reuse of the same in-memory segment; the probe did not inspect it). With the fix: 0.
• Two more trigger paths. f is new to the segment and the document is rejected because of another field: the first valid StringField f then fails with field 'f' cannot have docValuesSkipIndexType=RANGE with doc values type NONE. f exists only in an earlier committed segment: the index-wide check rejects the skip-indexed document, and 5 of 5 later StringField f documents fail before the next flush.
• All four factories (Sorted, SortedSet, Numeric, SortedNumeric indexedField) with StringField or StoredField as f's structure fail. Plain doc values without a skip index, points and vectors in the rejected document do not affect the next one.

• Who. Users of the doc-values skip index (SortedDocValuesField, SortedSetDocValuesField, NumericDocValuesField or SortedNumericDocValuesField indexedField, or FieldType#setDocValuesSkipIndexType) on a field name that some documents send with a skip-indexed doc value and others without doc values. One rejected document that carries the skip-indexed doc value is enough: rejected because f was established without doc values (in this or an earlier segment), or because another field of that document is invalid.
• What it costs. Every later document with f and no doc values in the same in-memory segment fails with IllegalArgumentException until the next flush, NRT reopen or commit (20,000 of 20,000 in the probe, where nothing flushed). Each failed document is deleted, as for any rejected document. Applications that log and drop failed documents lose valid data.
• Not affected. The writer stays open; in every exercised path the committed index, searches and other fields were fine; documents without f are accepted; no invalid document is accepted (by code reading, and none from this root cause in the study's bounded-exhaustive harness over its 11-type palette).
• Conditions. Default IndexWriterConfig; any codec (the rejection comes before any codec call: the test framework's per-seed codecs and, in the independent reproduction's test, the production default codec all fail); assertions on and off; ordinary scale; one thread. A second thread on the same writer was affected too (consistent with reuse of the same in-memory segment); with several concurrent segments, only the affected one should fail (inferred, not executed).
• How likely: rare. It needs the opt-in skip index on a field with a mixed per-document schema: a rejected document where f carries a skip-indexed doc value, then documents with f and no doc values. An application that always sends the same Lucene fields for a field name never produces that mix: either every document with f writes both doc-values members again (or, on the 10.5+ frozen-type fast path, keeps the already validated ones), or no document ever writes RANGE.

ByteBuffersDirectory and a plain IndexWriterConfig, so nothing flushes and nothing random affects the result: the test framework picks a codec per seed, but the rejection happens before any codec call. The test sits in org.apache.lucene.index, as Lucene's own IndexWriter tests do, but uses no package-private API. The expectThrows for doc 1 passes, so the correct rejection is unchanged; the third addDocument (line 42) throws, and that exception is exactly the wrong rejection, which is why the test has no extra assertion.

The other trigger paths (the field new to the segment, the field only in a committed segment) and the four indexedField factories share the root cause and were executed by the study's longer tests; they are not part of this repro.

Regression test: this same test; it passes with the fix. Upstream may prefer it as a method in TestPerFieldConsistency or TestIndexWriter.

cp $REPRO/TestRejectedSkipIndexDoc.java $LUCENE/lucene/core/src/test/org/apache/lucene/index/
cd $LUCENE && ./gradlew :lucene:core:test --tests TestRejectedSkipIndexDoc -Ptests.seed=DEADBEEF -Ptests.asserts=true
# assertions off fails the same way: -Ptests.asserts=false

# gradle, stock 645d4fe14a20, -Ptests.seed=DEADBEEF -Ptests.asserts=true (stack trimmed; -Ptests.asserts=false identical)
TestRejectedSkipIndexDoc > testValidDocAcceptedAfterRejectedSkipIndexDoc FAILED
    java.lang.IllegalArgumentException: Inconsistency of field data structures across documents for field [f] of doc [2]. doc values skip index type: expected 'NONE', but it has 'RANGE'.
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:AF01532DE0E3BA9D]:0)
        at org.apache.lucene.index.IndexingChain$FieldSchema.raiseNotSame(IndexingChain.java:2259)
        at org.apache.lucene.index.IndexingChain$FieldSchema.assertSame(IndexingChain.java:2254)
        at org.apache.lucene.index.IndexingChain$FieldSchema.assertSameSchema(IndexingChain.java:2350)
        at org.apache.lucene.index.IndexingChain.initAndValidateFields(IndexingChain.java:695)
        at org.apache.lucene.index.IndexingChain.processDocument(IndexingChain.java:641)
        ...
        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1492)
        at org.apache.lucene.index.TestRejectedSkipIndexDoc.testValidDocAcceptedAfterRejectedSkipIndexDoc(TestRejectedSkipIndexDoc.java:42)

# the three documents one by one (a small probe on the same main)
doc 0: accepted
doc 1: rejected: Inconsistency of field data structures across documents for field [f] of doc [1]. doc values type: expected 'NONE', but it has 'SORTED'.
doc 2: rejected: Inconsistency of field data structures across documents for field [f] of doc [2]. doc values skip index type: expected 'NONE', but it has 'RANGE'.

• The class says so. FieldSchema is "A schema of the field in the current document. With every new document this schema is reset" (IndexingChain.java:2210).
• The documented rule is about indexed documents. MIGRATE.md:1557-1564: "The per field data-structures are implicitly defined by the first document indexed that contains a certain field." Doc 2 matches the first indexed document exactly; doc 1 was never indexed. The rejection is non-aborting: the writer stays open for the next document.
• Upstream tests assert recovery. In its "diff segment, same index" phases, TestPerFieldConsistency starts a new in-memory segment (writer.flush() at :159, :207), adds documents that the index-wide check rejects, then, in that same in-memory segment, writer.addDocument(doc0); // add document with correct data structures and asserts it is indexed (:173, :225; asserts at :176-178, :228-230), the shape of the committed-segment trigger path described above; its random doc-values field never uses indexedField. The skip-index tests testDocValuesMixedSkippingIndex and testDocValuesSkippingIndexWithoutDocValues (TestIndexWriter.java:4998-5053) check only the rejected document and add nothing after it.
• The leftover pair is one Lucene forbids. (NONE, RANGE) is what FieldInfo.checkConsistency rejects (FieldInfo.java:160-168) and what updateDocFieldSchema rejects for a single field (IndexingChain.java:1566-1572).
• History shows an omission. LUCENE-9334 (d03662c48bf, 2021) created FieldSchema and reset every member it compared; LUCENE-10577 (a693fe819b0, 2022) added vectorEncoding to reset() in the same commit; #13449 (048770205cb) added the skip index member, its write, its comparison and its use in initializeFieldInfo, but no reset line, and #13784 (b59a357e586) kept it that way. Neither PR's review discusses reset().
• Asymmetry. The same three-document sequence with plain doc values (no skip index), points or vectors in doc 1 recovers at once.

The strongest case: the first document really is invalid, and IndexWriter does not promise to roll back every bit of transient state after a non-aborting exception; the DocumentsWriter javadoc says such exceptions "can still partially update the index structures. These updates are consistent, but, they represent only a part of the document" (DocumentsWriter.java:76-79).

Why it fails: that sentence is about index structures holding part of the failed document, which deleting the document makes invisible. The stale skip index type is not a structure any later document could match: it rejects documents identical to the first indexed one, the pair it leaves is one FieldInfo forbids, and whether a document is accepted then depends on when the last flush happened, which no documented rule describes. The works-as-designed review by an independent reviewer from another model family (OpenAI GPT-5.6), which ran its own tests, rated the argument moderate and the bug real.

Option A was built and run here and by every independent check. B and C were considered and not run.

One added line in IndexingChain.java, after :2336, so that the per-document reset clears both doc-values members that setDocValues writes together.

     void reset(int doc) {
       resetJustDocId(doc);
       omitNorms = false;
       storeTermVector = false;
       indexOptions = IndexOptions.NONE;
       docValuesType = DocValuesType.NONE;
+      docValuesSkipIndex = DocValuesSkipIndexType.NONE;
       pointDimensionCount = 0;

Trade-offs

• None found. setDocValues already writes the member whenever doc values are present (IndexingChain.java:2293-2295), so documents with doc values, with or without a skip index, are validated exactly as before.
• The frozen-type fast path (resetJustDocId, #15886) is untouched.
• Cost: one field store per field per document, on a path that already resets ten others.

What the checks validated

Applied to a scratch worktree of 645d4fe14a20 and run through gradle with OpenJDK 25.0.3, seed DEADBEEF (2026-10-30):

• stock: the minimal test fails with assertions on and off;
• with the fix: the minimal test passes with assertions on and off;
• the closest suites TestPerFieldConsistency, TestIndexWriter (incl. the two skip-index tests), TestIndexWriterExceptions, TestDocValuesIndexing, TestFieldInfos, TestDocValues and TestSortedSkipperScorerSupplier pass (213 tests, 0 failures, 4 skipped; assertions on, seed DEADBEEF);
• the whole :lucene:core:test passes (748 classes, 8,897 tests, 0 failures, 322 skipped; assertions on, seed DEADBEEF), and so does checkGoogleJavaFormat.

Every independent check also ran this line: the finder's 3 tests, the independent reproduction's 6, the intended-behaviour check's 7 and the TestPerFieldConsistency-pattern variant (5) pass with it; at the introducing commit its boolean form fixes the repro; in the bounded-exhaustive harness it removes all 1,404 wrongly rejected sequences.

Compare and pass docValuesSkipIndex only when docValuesType != NONE, in assertSameSchema and initializeFieldInfo.

Trade-offs

Hides the symptom at two read sites instead of fixing the state; any future reader of the member would see the stale value again. Not run.

Allocate a new FieldSchema per document, so no member can be forgotten.

Trade-offs

Adds an allocation per field per document on the indexing path; and reset() today never clears the attributes map (IndexingChain.java:2331-2343), which initializeFieldInfo passes on (:1405), so a wholesale reset changes more than this bug. Not run.

Method: semi-formal (the hunt's tier: the first signal was code reading while modelling, which a formal tool then witnessed).

• Model. An Alloy 6.2 model of the per-segment FieldSchema/PerField state machine, checked against a stateless reference of the per-document schema contract. The checks StockLocalNoPoison and RowOnlyStockNoPoison gave 3-document counterexamples, and StockNoPoison one at the full 18-schema scope; with the fix, SkipFixLocalNoPoison is UNSAT (a bounded proof in a small universe).
• Harness. A bounded-exhaustive differential harness on the real jar ran 246,848 document sequences over an 11-type palette and found 1,404 sequences where a valid document was rejected; this one-line fix alone removes all of them.
• Test. The counterexample was minimised to a JUnit on the stock jars; the minimal repro above is its first case, cut down to three documents and no extra assertion.