IndexWriter#addBatch rejects an untokenized column with positions that addDocument accepts

IndexWriter#addBatch (experimental, first in 10.5.0) rejects an untokenized BinaryColumn or DictionaryColumn whose field type indexes positions or term-vector positions. The internal BinaryColumnAdapter picks the single-term InvertableType.BINARY path from tokenized() alone (ColumnFieldAdapter.java:249 on main) and supplies no TokenStream; IndexingChain.invertTerm refuses proximity data on that path (IndexingChain.java:2136-2145). addDocument with new Field(name, BytesRef, sameType) indexes the value as one token at position 0, because Field always wraps an untokenized value in a single-token stream. ColumnValidation accepts the type first, and TokenStreamColumn refuses untokenized types, so no column kind can index it.

• Untokenized field: the whole value is one term, as in StringField. A FieldType may still ask for positions (DOCS_AND_FREQS_AND_POSITIONS) or term vectors with positions; Field(String, BytesRef, IndexableFieldType) accepts that and rejects only offsets and tokenized types (Field.java:209-215 on main).
• Invertable type: how the indexing chain turns a field into terms. TOKEN_STREAM reads a TokenStream; BINARY takes binaryValue() as a single term with no positions (InvertableType.java:25-32). Field is always TOKEN_STREAM and wraps an untokenized BytesRef in a one-token BinaryTokenStream (Field.java:459-487).
• Column batch API: IndexWriter#addBatch(ColumnBatch) (@lucene.experimental, 10.5.0) indexes columns instead of documents. package-info.java:35-43 lists term inversion among the uses of both BinaryColumn and DictionaryColumn, and the DictionaryColumn javadoc says "term inversion (tokenized or untokenized)" (DictionaryColumn.java:27-32). Internally each becomes a BinaryColumnAdapter (ColumnFieldAdapter.java:51-62), a Field subclass.

// ColumnFieldAdapter.java:244-262 (BinaryColumnAdapter)
return tokenized ? InvertableType.TOKEN_STREAM : InvertableType.BINARY;   // :249 looks at tokenized() only
...
return null;                                                              // :261 tokenStream(): none when untokenized

// IndexingChain.java:2136-2145 (invertTerm, the BINARY path)
if (fieldType.tokenized()
    || fieldType.indexOptions().subsumes(IndexOptions.DOCS_AND_FREQS_AND_POSITIONS)
    || fieldType.storeTermVectorPositions() || fieldType.storeTermVectorOffsets()
    || fieldType.storeTermVectorPayloads()) {
  throw new IllegalArgumentException("Fields that are tokenized or index proximity data must produce a non-null TokenStream, but " + ...);
}

1. IndexingChain.processBatch validates each column (:730-737). validateBinaryColumn and validateDictionaryColumn (ColumnValidation.java:172-249) never look at positions or term-vector options, so the type passes.
2. The field's options are registered (initAndValidateFields, :784), then the row pass (:793) creates the adapter. For an untokenized type it answers BINARY (:249).
3. invertTerm sees a type that indexes positions and throws (stack trace IndexingChain.java:2143). The message asks for a TokenStream; but TokenStreamColumn, the only column that supplies one, requires tokenized() == true (TokenStreamColumn.java:44-56).

The same happens for a DictionaryColumn (same adapter) and for a DOCS type with term-vector positions. Untokenized DOCS / DOCS_AND_FREQS columns, the common StringField-like case, and tokenized columns work.

• Who is hit. Applications that index through IndexWriter#addBatch with a BinaryColumn or DictionaryColumn whose untokenized type has DOCS_AND_FREQS_AND_POSITIONS or term vectors with positions, for example a schema they already index with addDocument.
• Default vs opt-in. Opt-in: the column API is experimental, and most untokenized fields (StringField, KeywordField) index DOCS or DOCS_AND_FREQS, which work.
• Effect. One IllegalArgumentException from the row pass, after validation accepted the column. The message points at a TokenStream the user cannot supply. The writer stays open and the failed batch adds no visible documents (a reader opened afterwards in the probe shows maxDoc=0). As after any non-aborting failure (a failed addDocument too), the batch's reserved doc IDs stay buffered as deleted docs and no flush is triggered (DocumentsWriterPerThread.java:339-354, :392-415).
• Scale. None: one document, one value, single thread, with assertions on and off.
• Likelihood: rare.

The test lives in org.apache.lucene.document.column, like the upstream column tests, and makes no random() call of its own; the result does not depend on the seed. It is the minimal test with its one comment line shortened to pass the 100-column format check. It belongs in lucene/core/src/test/org/apache/lucene/document/column/ and passes :lucene:core:check -x test (format, lint, forbidden APIs).

Regression test: this same test (with fix B, the same test with addBatch wrapped in expectThrows).

TestBinaryColumnUntokenizedPositions > testUntokenizedColumnWithPositions FAILED
    java.lang.IllegalArgumentException: Fields that are tokenized or index proximity data must produce a non-null TokenStream, but f did not
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:A2908C7053426C4]:0)
        at org.apache.lucene.index.IndexingChain$PerField.invertTerm(IndexingChain.java:2143)
        at org.apache.lucene.index.IndexingChain$PerField.invert(IndexingChain.java:1960)
        at org.apache.lucene.index.IndexingChain.invertAndStore(IndexingChain.java:1483)
        at org.apache.lucene.index.IndexingChain.processRowColumns(IndexingChain.java:916)
        at org.apache.lucene.index.IndexingChain.processBatch(IndexingChain.java:793)
        at org.apache.lucene.index.DocumentsWriterPerThread.updateBatch(DocumentsWriterPerThread.java:343)
        at org.apache.lucene.index.DocumentsWriter.updateBatch(DocumentsWriter.java:469)
        at org.apache.lucene.index.IndexWriter.updateBatch(IndexWriter.java:1580)
        at org.apache.lucene.index.IndexWriter.addBatch(IndexWriter.java:1538)
        at org.apache.lucene.document.column.TestBinaryColumnUntokenizedPositions.testUntokenizedColumnWithPositions(TestBinaryColumnUntokenizedPositions.java:72)

• The contract. DictionaryColumn.java:27-32: "... for stored binary or string fields, and for term inversion (tokenized or untokenized)." package-info.java:35-43 lists "term inversion" for BinaryColumn and DictionaryColumn; BinaryColumn.java:23-29: "stored/indexed binary or text fields". No javadoc, test, CHANGES or MIGRATE entry restricts the index options of untokenized columns.
• No column keeps the type untokenized. TokenStreamColumn rejects untokenized types (TokenStreamColumn.java:44-56; TestColumnBatchTokenStreamColumn.testConstructorRejectsNonTokenized); the workaround works only by changing the type to tokenized.
• Validation is the place for rejections. ColumnValidation already rejects, for example, a DictionaryColumn with BINARY doc values with a specific message (ColumnValidation.java:216-222).
• Parity is a stated goal. #16224 (Tim-Brooks): without TokenStreamColumn "anyone using user provided token streams would need to fall back to the addDocument path"; #16708 (mayya-sharipova) fixed that "addBatch accepted vectors that addDocument would reject".
• History. The adapter line is unchanged since #15990 created the column API; its description, 7 comments, 11 reviews and 20 inline review comments never discuss positions on untokenized columns. #16101 lists "inverted tokenized/untokenized fields" among its tested uses.

The strongest case: InvertableType.BINARY deliberately carries no proximity data, and TestDocumentWriter.testIndexBinaryValueWithoutTokenStream (TestDocumentWriter.java:435, from #12139) asserts that a BINARY field with positions or term-vector positions is illegal ("cannot index positions on a binary field"). Untokenized columns could be meant as plain keywords, with positions left to TokenStreamColumn, and the API is experimental.

Why it fails: that test covers a field that declares BINARY itself; here the user never chooses BINARY, the internal adapter does, from tokenized() alone. Nothing documents a keyword-only reading, and TokenStreamColumn refuses untokenized types, so the workaround (a one-token TokenStreamColumn) means changing the type to tokenized. Even if maintainers decide the type is unsupported, the error belongs in ColumnValidation with a message that says so (fix B). The cross-model reviewer (OpenAI GPT-5.6) rated this defence moderate and not convincing.

Validated with gradle in a clean scratch worktree of 5adff78bb168 (OpenJDK 25.0.3, seed DEADBEEF, 2026-11-03). The suites: the six TestColumnBatch* classes in org.apache.lucene.document.column, TestColumnBatchIndexing and TestDocumentWriter (182 tests).

Workaround until fixed: a TokenStreamColumn with the same index options but tokenized=true, whose stream emits exactly the value's bytes as one token, gives the same index options, norms setting and postings (doc 0, freq 1, position 0; executed). A TokenStreamColumn is inverted-only (TokenStreamColumn.java:29-33): a stored value, doc values or points go in a separate column with the same name. For a multi-valued document it matches the Field(BytesRef) row path only while the analyzer's position increment gap is 0, the default (read from code, not executed).

In BinaryColumnAdapter, compute once whether the type is untokenized but indexes positions or term-vector positions, offsets or payloads (the condition invertTerm checks). Then invertableType() returns TOKEN_STREAM and tokenStream() delegates to Field.tokenStream, which wraps binaryValue() in its one-token stream. DOCS / DOCS_AND_FREQS columns keep the BINARY path.

     this.indexed = fieldType.indexOptions() != IndexOptions.NONE;
+    this.singleTokenStream =
+        tokenized == false
+            && (fieldType.indexOptions().subsumes(IndexOptions.DOCS_AND_FREQS_AND_POSITIONS)
+                || fieldType.storeTermVectorPositions()
+                || fieldType.storeTermVectorOffsets()
+                || fieldType.storeTermVectorPayloads());
...
-    return tokenized ? InvertableType.TOKEN_STREAM : InvertableType.BINARY;
+    return tokenized || singleTokenStream ? InvertableType.TOKEN_STREAM : InvertableType.BINARY;
...
+    if (singleTokenStream) {
+      // Field wraps an untokenized binaryValue() in a single-token stream
+      return super.tokenStream(analyzer, reuse);
+    }

Trade-offs

About 15 lines in one internal class; addBatch then indexes what addDocument indexes. It also sends offsets types down the single-token stream, while Field(String, BytesRef, ...) rejects offsets; maintainers may prefer to leave offsets out here and reject them in ColumnValidation with Field's message. Not tested with offsets, nor for a DictionaryColumn, term-vector positions or a DENSE column. Using subsumes, not compareTo, keeps DOCS_AND_CUSTOM_FREQS (which sorts after the positional options) on the BINARY path (IndexOptions.java:36-73).

What the checks validated

Stock 5adff78bb168: the repro fails through gradle with assertions on and off; with fix A it passes, on and off. With fix A, the six TestColumnBatch* classes, TestColumnBatchIndexing and TestDocumentWriter: 182 tests, 0 failures. :lucene:core:check -x test (format, lint, forbidden APIs) with fix A and the test: passes. Earlier, the first version of this change (compareTo instead of subsumes) passed the repro (on, off), 71 column tests, and the repro over the 10.5.0 jars.

In validateBinaryColumn and validateDictionaryColumn, throw when the type is untokenized and indexes positions or term-vector positions, offsets or payloads, with a message that names the column and the alternative, e.g. BinaryColumn "f" is untokenized and cannot index positions, offsets or payloads (indexOptions=DOCS_AND_FREQS_AND_POSITIONS); use a tokenized TokenStreamColumn or addDocument.

Trade-offs

Small, and the error comes from validation, before the field's options are registered, with a message that says what to do. But addBatch still cannot index a type addDocument accepts, so the column javadocs would need to say so. It also rejects such a column when it holds no values, which addBatch accepts today.

What the checks validated

With fix B the repro (assertions on), which asserts indexing, fails as intended for this option: addBatch now throws the new message from ColumnValidation.validateBinaryColumn, called by IndexingChain.processBatch:733 before the row pass. The same 8 suites: 182 tests, 0 failures; :lucene:core:check -x test passes.

Method: semi-formal (a proof-obligation audit of a class's contract, witnessed by an executable repro).

• The obligation: for every field type a BinaryColumn or DictionaryColumn accepts for term inversion, addBatch inverts the value as Field with the same type would. BinaryColumnAdapter.invertableType decides from tokenized() alone, while invertTerm also depends on positions and term-vector options, so the obligation fails for untokenized proximity types.
• A one-test JUnit witnessed it on main with assertions on and off. An independent reproducer, an executed review by another model family (OpenAI GPT-5.6) and a devil's-advocate verifier each wrote and ran their own test.