An index-sort field without doc values passes addDocument, then the flush closes the IndexWriter

Index sorting (IndexWriterConfig#setIndexSort) sorts each new segment by the sort field's doc values when the segment is flushed. Since LUCENE-8441 (7.5.0) addDocument rejects a sort field whose doc-values type does not fit the sort, and the writer stays usable. That check runs only when the field has doc values (IndexingChain#initializeFieldInfo, s.docValuesType != DocValuesType.NONE), so a document that indexes the sort field without doc values, for example only as a StringField, a point or a stored field, is accepted when it is the first document in the index with that field. From then on, correct documents for the field are rejected, because the field's schema says "no doc values". At the next flush the sorter reads the field through DocValues.getSorted / getNumeric / ..., which throws IllegalStateException: unexpected docvalues type NONE ... Re-index with correct docvalues type.; an exception during flush is aborting, so the IndexWriter closes with a tragic exception and every document added since the last commit is lost, valid ones included. Committed data stays intact and a new writer starts clean. Every built-in sort type and BinarySortField fail the same way, with assertions on or off. The flush failure dates from LUCENE-9330 (8.6.0); the missing check is older (6.2.0), with a different symptom before 8.6.0. Running the existing check for NONE too fixes it; the whole lucene/core test suite passes with that change.

• Index sort: IndexWriterConfig#setIndexSort(Sort), "the Sort order to use for all (flushed and merged) segments" (IndexWriterConfig.java:504). The built-in sorters read the sort field's doc values, a per-document column: a STRING sort reads SortedDocValues, LONG/INT/FLOAT/DOUBLE read NumericDocValues (SortField.java:569-600).
• A field without doc values: a StringField, a point such as LongPoint, or a StoredField is indexed or stored but has doc-values type NONE. A field used for both keyword search and sorting often needs two Field instances; for a STRING sort, a StringField plus a SortedDocValuesField.
• Flush: writing the buffered documents as a new segment (on commit(), an NRT reopen, a full RAM buffer or close()). An exception during flush is aborting: the IndexWriter closes itself with a tragic exception and rolls back to the last commit.

1. When a field first appears in the in-memory segment, IndexingChain#initializeFieldInfo checks it against the index sort with validateIndexSortDVType, but only if (indexWriterConfig.getIndexSort() != null && s.docValuesType != DocValuesType.NONE) (IndexingChain.java:1382-1385). The check would reject NONE (IndexingChain.java:1675-1686); it is never asked.
2. At flush, maybeSortSegment builds each sorter's comparator on the flush-time doc-values reader (IndexingChain.java:283-294). That reader returns null for a field whose FieldInfo is not of the requested type (IndexingChain.java:209-218), and its field infos list the field with type NONE (:244-247).
3. The sorter reads through DocValues.getSorted (SortField.java:572-573; numeric sorts through getNumeric). It gets null and calls checkField, which throws IllegalStateException because the field exists (DocValues.java:213-230, :281-289).
4. DocumentsWriterPerThread#flush treats it as aborting and fires a tragic event (DocumentsWriterPerThread.java:585-602); the commit's finally rolls the writer back (IndexWriter.java:3882-3883, :6098-6106).
5. Reach: since 9.0 a field has one schema per index. When the field already has doc values, a document without them does not close the writer. If a committed segment has them, addDocument rejects it (on 9.0-10.5.0 the commit then hits the separate #15125 NPE). If only the in-memory segment has them, it depends on the order of the earlier correct document's two instances: with the SortedDocValuesField first it is rejected; with the StringField first, the frozen-FieldType validation cache (#15886, 10.5.0; IndexingChain.java:621-637, :1876-1898) skips the check and the document is accepted as a missing value, on main and 10.5.2 (a separate finding, LFV-118). So this flush failure needs the document to be the first one in the index with the field (or the first after IndexWriter.deleteAll()), and then later correct documents are rejected until the flush fails.

• The brand example (main, assertions on and off): lamp committed, mug buffered, chair with brand only as a StringField accepted; sofa with StringField plus SortedDocValuesField rejected ("Inconsistency of field data structures across documents for field [brand] ... doc values type: expected 'NONE', but it has 'SORTED'."); commit() throws unexpected docvalues type NONE for field 'brand' (expected=SORTED); isOpen() false; a later addDocument throws AlreadyClosedException; the index holds 1 document, lamp. A second writer then commits mug and sofa: 3 documents.
• The repro test: doc 0 committed, doc 1 buffered, doc 2 with sort only as a StringField (STRING sort) or a LongPoint (LONG sort): the commit throws, the writer closes, the index holds 1 document.
• Executed for every built-in sort type (STRING, LONG, INT, FLOAT, DOUBLE, SortedNumericSortField, SortedSetSortField) and BinarySortField.

• Who is exposed. Applications with an index sort (the default IndexWriterConfig has none) whose first document with the sort field indexes it without doc values: the SortedDocValuesField (or NumericDocValuesField, ...) next to the StringField or point is missing, or the field is only stored. Executed for STRING (StringField), LONG, INT, FLOAT and DOUBLE (points), SortedNumericSortField (StoredField), SortedSetSortField (StringField) and BinarySortField (StoredField).
• What it costs. The writer closes with a tragic exception and every document since the last commit is lost, including documents accepted before the bad one; later calls throw AlreadyClosedException. Committed segments are intact and a new writer starts clean. Until the flush, correct documents for the field are rejected too.
• Default vs opt-in. Index sorting is opt-in. Codec, directory and assertions do not matter: the plain probe uses the default codec and ByteBuffersDirectory, the JUnit tests the test framework's randomized codec and directory, and all fail with assertions on and off.
• Scale. One document; deterministic at the first flush that contains it. In the repros that flush is a commit(); an NRT reader open, a flush triggered inside addDocument or close() runs the same code, so the failure would surface there instead (read, not executed).
• How likely. Rare: a schema mistake that fails loudly with a clear message at the first flush, so it is usually caught in development, and no public report was found in the documented, best-effort search. The point is the disproportion: Lucene already rejects the neighbouring mistake, a wrong doc-values type, cleanly at addDocument.
• Before 8.6.0 (6.2.0 to 8.5.2, end of life) the document was also accepted, the segment was committed, and in the probes a later sorted search and forceMerge(1) failed, the merge closing the writer in a later session too.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: accepted by addDocument, then commit threw java.lang.IllegalStateException: unexpected docvalues type NONE for field 'sort' (expected=SORTED). Re-index with correct docvalues type.; writer open=false
       org.junit.Assert.fail(Assert.java:89)
       org.apache.lucene.index.TestIndexSortFieldWithoutDocValues.testSortFieldIndexedOnlyAsStringField(TestIndexSortFieldWithoutDocValues.java:45)

• Contract. LUCENE-8441, CHANGES 7.5.0: "IndexWriter now checks doc value type for index sort fields and fails the document if they are not compatible." Its issue (#9487), @jimczi: "Adding a doc value of the wrong type will now throw an exception when the document is indexed instead of failing at commit time." setIndexSort: "Set the Sort order to use for all (flushed and merged) segments", nothing more. IndexWriter class javadoc: "if you hit an Error, or disaster strikes during a checkpoint then IndexWriter will close itself"; getTragicException: closed "as a side-effect of a tragic exception, e.g. disk full while flushing a new segment". Here one document's field type triggers that tragic path.
• Tests. TestIndexSorting#testWrongSortFieldType asserts the up-front IllegalArgumentException for 4 sort types times 3 wrong doc-values types, with no case without doc values; no test asserts the flush-time message.
• History. From 6.5.0 (LUCENE-7579, flush-time sorting) to 8.5, maybeSortSegment skipped a sort field without doc values (// safe to ignore, sort field with no values or already seen before). LUCENE-9330 (#1440, 2020) replaced it with a sorter that reads through DocValues.getX; neither the issue nor the PR mentions fields without doc values. LUCENE-8441 checked only where a document added doc values; before 9.0 a document without them simply had a missing value. LUCENE-9334 (9.0) moved the check and kept != NONE, which now means "this field has no doc values in the schema". No built-in sorter can read such a field, which fits LUCENE-8441's broad CHANGES entry, though its issue speaks of "a doc value of the wrong type" and no commit records whether NONE was meant to be in scope.
• Precedent. Upstream fixed a sibling failure in the same flush-time reader as a bug: #15125, "Handle inconsistent schema on flush with index sorts" (10.5.1).

The strongest "intended" case: index sorting is doc-values based; DocValues.getX documents IllegalStateException "if field exists, but was not indexed with docvalues" and the message says what to do; since 9.0 the mistake is static and fails loudly at the first flush instead of silently giving an unsorted index; LUCENE-8441 spoke of a doc value "of the wrong type"; exceptions during flush are aborting by design.

Why it is weaker than it looks (our reading; the call is the maintainers'): that javadoc describes a reader-side helper, not a promise that the writer closes and drops other documents; the writer close arrived with a refactoring that removed an explicit "safe to ignore" branch, and neither its issue nor its PR discusses this case, so nothing records the close as a design decision; the NONE exemption dates from pre-9.0 schema rules, no built-in sorter can read such a field, and validateIndexSortDVType already handles NONE (a custom IndexSorter might cope with it); upstream treats a document-caused flush failure in this reader as a bug (#15125). What stands: it is a misconfiguration, caught early in practice, so the priority is low to medium. The adjudication confirmed it with medium confidence for that reason; the devil's advocate rated "not intended" medium-high. A read-only cross-model fact check rated "documented misuse" and "fix A is not the only right fix" moderate: maintainers could agree it is a bug and still choose another resolution, such as a check limited to built-in sorters, an explicit sorter contract, or documentation.

The rejection uses the message style of testWrongSortFieldType ("SortField <string: \"sort\"> expected field [sort] to be SORTED but it is [NONE]"), and the other buffered documents are committed. Fields that are not sort fields still pass: the validation reader throws only for the field being validated. Until a fix ships, give every sort field doc values from its first document, or validate documents before addDocument; committed data is not affected, so no reindex is needed. Either way, add a case without doc values to testWrongSortFieldType (or the new test). Treating the field as missing is not recommended. Done only in the flush-time reader, the field would reach a committed segment, where merges and sorted searches read it through the same strict helpers. Done in DocValues.checkField, it would change every caller of DocValues.getX, which upstream declined in #11504.

IndexingChain.java:1382: one condition removed.

     FieldSchema s = pf.schema;
-    if (indexWriterConfig.getIndexSort() != null && s.docValuesType != DocValuesType.NONE) {
+    // Also check a field without doc values (NONE): built-in index sorters cannot read it at flush.
+    if (indexWriterConfig.getIndexSort() != null) {
       final Sort indexSort = indexWriterConfig.getIndexSort();
       validateIndexSortDVType(indexSort, pf.fieldName, s.docValuesType);
     }

Trade-offs

• The check now also runs for fields without doc values, once per field per in-memory segment and only when an index sort is configured; it builds comparators over maxDoc 0. Not benchmarked.
• A custom IndexSorter that reads the field directly and copes with a missing value at flush would now see the document rejected. Built-in sorters cannot sort such a field anyway.

What the checks validated

• Scratch worktree of a9779ef3c193, gradle, JDK 25: the new test fails 2 of 3 on stock with assertions on and off; with the fix, the new test plus TestIndexSorting, TestBinarySortField, TestSortingCodecReader and TestIndexWriter pass (5 suites, 212 tests, 1 skipped); the whole :lucene:core:test passes with assertions on (749 suites, 8,892 tests, 320 skipped, 0 failures) and off (326 skipped, 0 failures); every other module test whose own source calls setIndexSort (misc, highlighter, backward-codecs, sandbox: 9 suites, 165 tests) passes (17 more non-core classes that inherit index-sort cases from test-framework base classes were not run with the fix); :lucene:core:check -x test (google-java-format, ecj lint, forbidden APIs) passes.
• Standalone runner with the fix shadowed: the new test 3 of 3 with assertions on and off; the session probe commits 3 documents. Earlier, the finder's equivalent sketch made the finder's test, the independent reproducer's 9 tests and its BinarySortField test pass, with TestIndexSorting and TestBinarySortField (87 tests) still green.
• The code comment was reworded afterwards (comment only: "built-in index sorters cannot read it"). Re-checked with the reworded file: the diff applies to a9779ef3c193 and yields the patched source exactly; the new test passes 3 of 3 with assertions on and off; the session probe commits 3 documents; :lucene:core:check -x test passes.

Two different changes. B1: only the flush-time reader treats a field with doc-values type NONE as having no values, as 6.5.0-8.5.2 did at flush. B2: DocValues.checkField becomes lenient for such a field.

Trade-offs

Not validated, not recommended (read, not executed). B1: the segment would be committed with the field still of type NONE, and merge-time sorting (SortField.java:569-600) and sorted search (TermOrdValComparator.java:165, NumericComparator.java:147, FieldComparator.java:273) read through DocValues.getX, so they would likely fail later, as on 6.2.0-8.5.2. B2: those reads would return empty values instead, but every caller of DocValues.getX would change; upstream kept checkField strict in #11504 (LUCENE-10468), where @jpountz asked: "But if it exists and is not indexed correctly, it's better to fail?"

Index the sort field with doc values from its first document (for example StringField plus SortedDocValuesField, or LongPoint plus NumericDocValuesField), or validate documents before addDocument.

Trade-offs

Avoids the failure; not a fix.

Method: non-formal (code reading). The hunt unit modelled the flush-time sort for BinarySortField, whose key provider reads through DocValues.getBinary on the flush-time reader. Reading DocValues.checkField showed that it throws for a field that exists without doc values, and IndexingChain#initializeFieldInfo showed that such a field is never checked up front.

• A direct probe on main confirmed it for STRING, LONG and BinarySortField sorts; a public-API JUnit test reproduced it with assertions on and off, and a version sweep on release jars located the change of symptom at 8.6.0.
• The unit's bounded-exhaustive harness did not find it: its alphabet always indexed the sort field with doc values or not at all.
• Then an independent reproducer, read-only reviews by another model family (OpenAI GPT-5.6), a devil's advocate, an adjudication, the follow-up stages and an audit each checked it (see the checks below).