DatasetSplitter.split with explicit fieldNames copies only the first value of each multi-valued field into the split indexes

DatasetSplitter.split copies the documents of an index into a training, a test and a cross-validation index for classifier work. Its fieldNames parameter lists the "names of fields that need to be put in the new indexes or null if all should be used". With null it copies every stored text, string and numeric value; with names it calls document.getField(fieldName) once per name, and Document.getField returns only the first value when several fields share the name. So every value after the first of a named multi-valued field is silently dropped, from the stored fields and from the postings of all three output indexes, because each output field is re-indexed from the copies. On 3 documents with two text values each, naming every stored field leaves text = [first value] and docFreq(text:second) = 0 in each output index, where null keeps both values and 1. It needs the opt-in classification module and an explicit name list, is in every release from 4.2.0 to 10.6.0 (10.6.0 by reading) and main, and looping over getFields(fieldName) fixes it.

• Split: new DatasetSplitter(testRatio, crossValidationRatio).split(reader, training, test, cv, analyzer, termVectors, classFieldName, fieldNames...) groups the source documents by class and sends each one to the test, cross-validation or training index (DatasetSplitter.java:80-180).
• fieldNames: "names of fields that need to be put in the new indexes or null if all should be used" (:76-77).
• Multi-valued field: a document may hold several fields with the same name ("A Document is a set of fields", Document.java:31). getField(name) returns the first one, getFields(name) all of them (Document.java:141-161).

1. Every output field gets the same type, new FieldType(TextField.TYPE_STORED) (DatasetSplitter.java:129).
2. createNewDoc reads the stored source document (:186). With a non-empty fieldNames (:187) it takes, for each name, IndexableField field = document.getField(fieldName) (:189) and adds new Field(fieldName, field.stringValue(), ft) (:191).
3. Document.getField: "If multiple fields exists with this name, this method returns the first value added" (Document.java:142-143). The other values are never read.
4. With null or no names it loops over document.getFields() (:195) and copies every value (:196-204); a stored binary field makes that branch throw instead (:198-199, Field.java:213-214).
5. So in named mode each output document has one value per named field. The output index is built from these copies, so the dropped values are missing from the stored fields and from the postings of the test, cross-validation and training indexes alike.

Source: 3 documents d0, d1, d2, each with id, text = "first value" + "second value" (stored TextFields) and class cls = "a" (stored, plus sorted doc values). new DatasetSplitter(0.3, 0.3) sends d0 to test, d1 to cross-validation and d2 to training (executed, stock main):

At a larger size (the security screen's probe: 257 live documents in 12 segments), naming every field kept 257 of 641 body values and 257 of 342 acl values, one per document; null kept all of them.

• Who is exposed. Callers of DatasetSplitter.split with a non-empty fieldNames list. Passing names is the module's own pattern: Test20NewsgroupsClassification (a @Monster test) splits with CATEGORY_FIELD, BODY_FIELD, SUBJECT_FIELD, CATEGORY_FIELD, with single-valued fields, so it is not hit itself. No production code in Lucene calls the class.
• What triggers it. Any named field with more than one stored value. One document with two values shows it; every such document loses values, deterministically.
• What breaks. The split indexes, and so classifiers trained or evaluated on them, see fewer values than the source: in the probe of 257 documents about 60% of the body values (384 of 641) were gone. Silent: no exception, CheckIndex clean on all 9 output indexes of the probe.
• Not affected. The source index (its contents; split() does close the reader it is given), null or empty fieldNames, single-valued fields, and which document goes to which index.
• How likely. Certain once a named text, string or numeric field has several values, but the utility has few users, the loss only shows as weaker training data, and no public report was found (104 logged queries).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<[first, second]> but was:<[first]>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:120)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.classification.utils.TestDatasetSplitterMultiValuedField.testNamedFieldKeepsAllValues(TestDatasetSplitterMultiValuedField.java:53)

• Contract. "@param fieldNames names of fields that need to be put in the new indexes or null if all should be used" (DatasetSplitter.java:76-77). It selects fields by name against "all"; it is implied, not stated, that a named field keeps all its values.
• Symmetry. The null branch of the same method copies every value (:194-205); naming every field of a document gives a smaller result than null for the same document.
• Tests. TestDataSplitter adds one value per field and checks only maxDoc per output; its testSplitOnSomeFields names id and text. No test uses a multi-valued field with fieldNames, and the fix keeps the whole module suite green.
• History. The named branch has read getField(fieldName) since the first commit (3459b28334e, LUCENE-4345, 2012), next to an else branch that looped over getFields(); LUCENE-4345 never discusses multi-valued fields. No javadoc, comment, CHANGES or MIGRATE entry records a first-value rule.
• Precedents. LUCENE-5833 (2014, DocumentDictionary, CHANGES "New Features": it "now enumerates each value separately in a multi-valued field (not just the first value)") and SOLR-8871 (96489d23846, 2016, KNearestNeighborClassifier, part of a set of classification improvements) made the same change.

The strongest "intended" case (the cross-model reviewers rated it moderate; the devil's-advocate verifier rejected it): Document.getField documents first-value semantics, so the code does what it calls; the splitter is lossy anyway (every copied value is rebuilt as a stored TextField, doc values and points are dropped); the line is unchanged since 2012; some classifiers in the module read only the first value; and both precedents went in as a feature or an improvement, not as bug fixes.

Why it fails: the lossiness both modes share changes field types, never values; named mode alone drops values, from the stored fields and the postings (the verifier's 3-valued field: tags:beta matches 0 of 6 training documents named, 6 of 6 with null). The javadoc sets names against "all should be used", so naming every field should equal null. Here the full behaviour already exists in the same method, while the precedents added it where it did not. What a classifier reads is a separate choice; the splitter decides what data exists at all. Every reviewer rated the bug real, and the adjudication confirmed it with high confidence; a maintainer may still label the fix an improvement.

One fix was built and validated in a scratch worktree of a9779ef3c193; the alternatives are listed with their trade-offs.

In createNewDoc (DatasetSplitter.java:189), iterate document.getFields(fieldName) instead of taking document.getField(fieldName). A name the document lacks still adds nothing, because getFields returns an empty array. The hunt's own fix is the same change.

     if (fieldNames != null && fieldNames.length > 0) {
       for (String fieldName : fieldNames) {
-        IndexableField field = document.getField(fieldName);
-        if (field != null) {
+        for (IndexableField field : document.getFields(fieldName)) {
           doc.add(new Field(fieldName, field.stringValue(), ft));
         }
       }

Trade-offs

• Named mode then copies every value of each named field, in order, like null mode; the repro's three indexes hold both values and docFreq(text:second) = 1 each.
• No change in meaning was found for text and string values, the only kind the fix was run with. Output size and indexing work grow with the values now kept (the security probe's two 20,000-value documents: 40,000 values copied instead of 2).
• A named field whose first value is text and a later value binary is copied (text only) today; with the fix the binary value is reached and split() throws (Field.java:245). Binary, numeric and mixed-type values and termVectors = true were not run with the fix.
• It keeps two limits that both modes share and that are not this bug: a stored binary field makes split() throw (named mode: value must not be null; null mode: cannot set a BytesRef value on a tokenized field), and numeric fields are copied as text.

What the checks validated

• Gradle, JDK 25, scratch worktree of a9779ef3c193: the new test fails 1 of 2 on stock with assertions on and off; with the fix, the new test plus TestDataSplitter pass (2 suites, 4 tests), the whole :lucene:classification:test passes with assertions on and off (13 suites, 45 tests, 1 skipped, 0 failures each), and :lucene:classification:check -x test passes.
• Standalone runner with the fix shadowed: the new test 2 of 2 (-ea, -da), the finder's test 1 of 1, the devil's-advocate verifier's test 3 of 3.
• Earlier, in the hunt: the 42 upstream classification tests with the same change.

Move the per-value copy of the null branch (DatasetSplitter.java:196-204) into a helper and call it from both branches, the named one looping over getFields(fieldName).

Trade-offs

The two modes share one code path and cannot drift again, but it is a larger diff. Numeric fields come out the same (both branches copy them as text); a named binary field would still make split() throw, only with the null branch's message (cannot set a BytesRef value on a tokenized field instead of value must not be null).

Add "only the first value of each named field is copied" to the fieldNames javadoc.

Trade-offs

No behaviour change: the silent loss and the difference from null mode stay.

Pass null (or no) field names: every value of every stored text, string or numeric field is copied; a stored binary field in the source makes this mode throw. Or store multi-valued content as one value per field before splitting. The source index is untouched, so re-running the split after a fix is enough, into new or empty output directories: split() opens its writers with the default OpenMode.CREATE_OR_APPEND (DatasetSplitter.java:92-94, LiveIndexWriterConfig.java:135), so re-using the old ones appends to the faulty output.

Trade-offs

null also copies stored fields you did not want.

Method: semi-formal (a bounded-exhaustive check of the real code), then a JUnit test. A hunt unit (classification k-NN and utilities against reference semantics), 2026-10-27, main a9779ef3c193.

• The finder's checker enumerated 1 to 4 documents (class none, a or b; 1 or 2 values of a stored text field; one segment or one per document; no deletion or the first document deleted; fieldNames null or {id, text, cls}; ratios 0.1 / 0.1 or 0.5 / 0.25) and ran the real split on each, checking among others C2: each output document has the requested stored fields with all of their values.
• Over all indexes of 1-4 documents (the checker's counters run on across sizes): 3,392 field mismatches in 24,816 splits; 4 documents alone: 2,696 in 20,736. All are in named mode with a 2-valued field, none in null mode; splits that threw or lost a document are counted separately. Example: expected text=[t0v0, t0v1], got text=[t0v0].
• The finding was reduced to a public-API JUnit on the stock main jars that fails with assertions on and off, and on 10.5.2 and 9.12.3.