DatasetSplitter.split with no fieldNames throws at the first document that has a stored binary field

DatasetSplitter.split copies the documents of an index into a training, a test and a cross-validation index for classifier work. Its fieldNames parameter lists the "names of fields that need to be put in the new indexes or null if all should be used". In the all-fields mode it copies every stored value, and it rebuilds each copy with one field type, new FieldType(TextField.TYPE_STORED): indexed, tokenized and stored. For a binary stored value it calls new Field(name, bytes, ft), which core Field rejects with "cannot set a BytesRef value on a tokenized field". split() wraps that in an IOException at the first copied document that stores a binary value, such as new StoredField("payload", new byte[] {1, 2, 3}), so in that mode such an index cannot be split. The branch was written for binary values in the very first version (2012) and has never worked: every release from 4.2.0 to 10.6.0 and main fails, only the message changed. It is loud and deterministic, needs the opt-in classification module, leaves the source index alone, and naming only the non-binary fields avoids it. Copying the bytes as a stored-only StoredField fixes it.

• Split: new DatasetSplitter(testRatio, crossValidationRatio).split(reader, training, test, cv, analyzer, termVectors, classFieldName, fieldNames...) groups the source documents by class and sends each one to the test, cross-validation or training index (DatasetSplitter.java:80-180).
• fieldNames: "names of fields that need to be put in the new indexes or null if all should be used" (:76-77). Leaving it out (or passing null or an empty array) is the all-fields mode.
• Stored binary field: new StoredField(name, byte[]) or new StoredField(name, BytesRef), "a stored-only field with the given binary value" (StoredField.java:65-77, :95-107). Reading the document back gives a field whose binaryValue() is the bytes.
• Tokenized field: a field whose value goes through the analyzer into terms. TextField.TYPE_STORED is indexed (DOCS_AND_FREQS_AND_POSITIONS), tokenized and stored (TextField.java:40-43). Bytes cannot be analyzed, so core forbids a BytesRef value on such a type.

1. split builds one type for every copied field: FieldType ft = new FieldType(TextField.TYPE_STORED), plus term vectors with positions and offsets if termVectors is true (DatasetSplitter.java:128-134).
2. For each source document it calls createNewDoc (:150). With no names (:187) it loops over every stored field (:195) and picks by value kind: reader (:196-197), binary (:198-199), string (:200-201), numeric as text (:202-203).
3. The binary branch adds new Field(field.name(), field.binaryValue(), ft) (:199). The constructor throws "cannot set a BytesRef value on a tokenized field" for an indexed, tokenized type (Field.java:213-214). Its javadoc states a broader rule, "@throws IllegalArgumentException if ... the field's type is indexed()" (:196-197); the code accepts an indexed type that is not tokenized, as in StringField(String, BytesRef, Store). ft is tokenized, so it is rejected either way.
4. split catches it and throws new IOException(e) (DatasetSplitter.java:174-175), at the first document with a stored binary value.
5. Same cause, other messages: with termVectors = true the offsets check fires first, "It doesn't make sense to index offsets on binary fields" (Field.java:209-211, executed); naming the binary field in fieldNames gives "value must not be null", because stringValue() is null for bytes (DatasetSplitter.java:191, Field.java:244-245, executed).

The hunt's check over six stored-value kinds and both copy modes, on stock main and with the hunt's fix (B, executed). A cell is OK when split() returns and exactly one document arrives across the three outputs; the check does not compare the copied values (numbers are rebuilt as text). With the recommended fix A, the named binary cell still throws (executed by the fact check).

The repro uses the first binary cell: one document, fieldNames left out.

• Who is exposed. Callers of DatasetSplitter.split in its all-fields mode (fieldNames omitted, null or empty) on an index where any document has a stored binary field. No production code in Lucene calls the class; only TestDataSplitter and Test20NewsgroupsClassification do.
• Default vs opt-in. The all-fields mode is the documented "all should be used" case and is tested upstream (testSplitOnAllFields). No index setting or scale is needed. On main the exception is thrown while the copy is built, before it reaches a writer, so neither the codec nor the analyzer plays a part; the test uses a plain IndexWriterConfig with the test framework's MockAnalyzer and randomized codec.
• What breaks. In that mode the call throws at the first document it copies that stores a binary value, deterministically, with assertions on and off, so the split never completes. Documents copied before that one stay in the outputs, because the writers are closed, and so committed, in finally (DatasetSplitter.java:178): a fact-check probe found the training index holding 1 document after the failure when a stored-string document came first. In the one-document repro no document reaches any output.
• Not affected. The source index (only read; split() does close the reader it is given), stored text, string and numeric values, and named mode that leaves the binary fields out (the cross-model reviewer ran it: passes).
• How likely. Certain once a stored binary field meets the all-fields mode, but the utility has few users, text-classification corpora mostly store text, the failure is loud with an obvious workaround, and no public report was found in 13 years.

The class extends LuceneTestCase and sits in DatasetSplitter's package next to TestDataSplitter; nothing package-private is used. It uses a plain IndexWriterConfig and MockAnalyzer, passes :lucene:classification:check -x test and runs unchanged on 10.6.0, 10.5.2 and 9.12.3. The failure does not depend on the seed (the exception comes from the field type, not from the randomized test setup); keep the seed for reproducibility.

The last assertion checks that the 3 bytes arrive, which is what the suggested fix gives. If a maintainer prefers to skip binary fields instead, only that assertion changes.

cp $REPRO/TestDatasetSplitterStoredBinaryField.java $LUCENE/lucene/classification/src/test/org/apache/lucene/classification/utils/
cd $LUCENE && ./gradlew :lucene:classification:test --tests TestDatasetSplitterStoredBinaryField -Ptests.seed=5EED1071 -Ptests.asserts=true
# also with -Ptests.asserts=false; the failure does not depend on the seed (keep it for reproducibility)

# gradle, stock main 645d4fe14a20, -Ptests.asserts=true (the same with false); left out: the seed frame and the echoed test output before the last line
TestDatasetSplitterStoredBinaryField > testSplitAllFieldsWithStoredBinaryField FAILED
    java.io.IOException: java.lang.IllegalArgumentException: cannot set a BytesRef value on a tokenized field
        at org.apache.lucene.classification.utils.DatasetSplitter.split(DatasetSplitter.java:175)
        at org.apache.lucene.classification.utils.TestDatasetSplitterStoredBinaryField.testSplitAllFieldsWithStoredBinaryField(TestDatasetSplitterStoredBinaryField.java:48)
        Caused by:
        java.lang.IllegalArgumentException: cannot set a BytesRef value on a tokenized field
            at org.apache.lucene.core@11.0.0-SNAPSHOT/org.apache.lucene.document.Field.<init>(Field.java:214)
            at org.apache.lucene.classification.utils.DatasetSplitter.createNewDoc(DatasetSplitter.java:199)
            at org.apache.lucene.classification.utils.DatasetSplitter.split(DatasetSplitter.java:150)
            ... 1 more
1 test completed, 1 failed

# the same failure through the standalone runner on main (-ea, -da), 10.6.0 (-ea, -da) and 10.5.2 (-da)
# 9.12.3 (-da): java.io.IOException: java.lang.IllegalArgumentException: Field must have either TokenStream, String, Reader or Number value; got stored,indexed,tokenized<payload:[1 2 3]>
# with the fix: 1 of 1 passes

• Contract. "@param fieldNames names of fields that need to be put in the new indexes or null if all should be used" (DatasetSplitter.java:76-77). The only stated precondition concerns the class field: "this must be indexed with sorted doc values" (:74-75). Nothing limits the kinds of stored fields in the source.
• The code's own intent. :198-199 is a branch for binary values, which strongly suggests they were meant to be copied, not refused (the branch alone cannot prove the author's intent).
• Core is right, the caller is not. Field documents the rejection (Field.java:196-197). The splitter makes that illegal call itself, for legal input.
• Not a write failure. split() declares "@throws IOException if any writing operation fails on any of the indexes" (DatasetSplitter.java:78); here nothing failed to write, the copy built an illegal field.
• Tests. TestDataSplitter indexes text fields and sorted doc values only; Test20NewsgroupsClassification stores StringField and TextField values. These are the only current tests that use the class, so no current upstream test stores a binary field for it, and none expects the exception; with the fix the whole module suite stays green.
• History. The branch is in the very first DatasetSplitter (3459b28334e, LUCENE-4345, 2012), when the constructor already refused an indexed BytesRef ("Fields with BytesRef values cannot be indexed"). LUCENE-4345 never discusses binary values; the recorded searches found no CHANGES or MIGRATE entry that records a limit.

The strongest "intended" case (the devil's-advocate verifier built it and rejected it; the cross-model reviewer rated it weak): the splitter is a text-classification helper, not a general index copier. It rebuilds every value as a stored TextField and turns numbers into text (DatasetSplitter.java:202-203), so binary values are out of scope; core documents the exception; split() declares IOException; and naming only the text fields works.

Why it fails: no documented limit was found, and the code's own binary branch points the other way. Core documenting the rejection puts the defect in the splitter, which builds the illegal field. The workaround does not make the documented "all" mode work: that mode can never succeed once a document it copies stores a binary field. The recorded searches found no test, issue, CHANGES or MIGRATE entry that treats the exception as expected. Every reviewer rated the bug real, and the adjudication confirmed it (Low).

One fix was built and validated in a scratch worktree of 645d4fe14a20; the hunt validated a two-branch variant with its standalone runner. The other options are listed with their trade-offs.

In createNewDoc's all-fields branch (DatasetSplitter.java:198-199), copy the bytes with new StoredField(field.name(), field.binaryValue()) (StoredField.java:105) instead of a Field with the shared tokenized type. Text, string, numeric and reader values are unchanged.

 import org.apache.lucene.document.FieldType;
+import org.apache.lucene.document.StoredField;
 import org.apache.lucene.document.TextField;
@@ createNewDoc, all-fields branch
         } else if (field.binaryValue() != null) {
-          doc.add(new Field(field.name(), field.binaryValue(), ft));
+          // a binary value cannot be indexed as text: copy it as a stored-only field
+          doc.add(new StoredField(field.name(), field.binaryValue()));
         } else if (field.stringValue() != null) {

Trade-offs

• The copy is stored, not indexed. The Document that StoredFields.document() returns carries no indexing metadata (keeping it would need a separate lookup of the reader's field infos), and bytes cannot be tokenized anyway. Core does allow an indexed, non-tokenized binary term (StringField(String, BytesRef, Store)), so a maintainer may prefer an explicit policy for binary values.
• StoredField has its own type, so the fix also works with termVectors = true (executed side probe: stock throws "It doesn't make sense to index offsets on binary fields", the fix passes). The binary copy then gets no term vectors (it is stored only), while the parameter says "true if term vectors should be kept".
• Named mode is left as it is: naming a binary field still throws "value must not be null" (DatasetSplitter.java:191), the line LFV-681's fix changes, so fix A covers the all-fields mode only (the 12-case check gives 11 of 12 with it). The two diffs touch different lines and apply cleanly with git apply in either order (executed).

What the checks validated

• Gradle, JDK 25, scratch worktree of 645d4fe14a20: the test fails 1 of 1 on stock with assertions on and off; with the fix, the test plus TestDataSplitter pass (2 suites, 3 tests), the whole :lucene:classification:test passes with assertions on and off (13 suites, 44 tests, 1 skipped @Monster, 0 failures each), and :lucene:classification:check -x test passes.
• Standalone runner with the fix shadowed: the test 1 of 1 (-ea, -da); the termVectors = true side probe 1 of 1.

Move the per-value copy into a helper that copies a binary value as new StoredField(name, bytes), and call it from both the named and the all-fields branch. It also fixes the named-mode binary case.

Trade-offs

A larger diff, and it rewrites DatasetSplitter.java:191, the line LFV-681's fix changes; it is the better shape for one change that fixes both bugs (together with LFV-681's getFields(fieldName) loop).

What the checks validated

By the hunt, with the standalone runner: the repro passes, upstream TestDataSplitter 2 of 2, the 12-case value-kind check 12 of 12; the independent reproducer's and the cross-model reviewer's own tests pass with it. Not run through gradle or the whole module.

Drop the binary branch, so binary stored fields are not copied.

Trade-offs

The split succeeds, but the bytes the caller asked for ("all should be used") are silently left out; the repro's last assertion would become "the document arrives".

Say in the fieldNames javadoc that stored binary fields must be left out by naming the other fields.

Trade-offs

No behaviour change; the dead branch and the crash stay.

Method: non-formal: code reading, then executed tests. A hunt unit (classification), 2026-10-26.

• The finder read createNewDoc's binary branch next to the shared TextField.TYPE_STORED type, wrote a JUnit, saw it fail on main, 10.5.2, 9.12.3, 9.5.0 and 9.4.2, and recorded it as a lead.
• The adjudication (2026-11-01) re-ran the lead, reduced it to one JUnit with one document, and ran a bounded differential: six stored-value kinds (string, binary, int, long, float, double) x both copy modes (fieldNames left out or naming the field), on stock main and with its fix (B). On main only the 2 binary cells throw; with fix B 12 of 12 pass (with the recommended fix A, 11 of 12).
• Independent verifiers, a cross-model reviewer, an adjudication and an audit then re-ran it (see the checks below).