DatasetSplitter can silently leave a whole labelled class out of its training, test and cross-validation indexes when some documents have no class value

DatasetSplitter.split copies an index into three new indexes (training, test, cross-validation) for classifier work. Since LUCENE-7196 (6.1.0) it groups the documents by the class field with GroupingSearch, so that the test and cross-validation shares are taken from each class separately, and it asks for as many groups as there are class values. Documents without a class value form one more group, the null group (the grouping default), which that count never includes. On a single segment that holds an unlabelled document the limit is one group short (unless a class value left only on deleted documents inflates the count), and the group whose first document comes last in index order is left out of all three outputs. With d0 (no class value), d1 (class a) and d2 (class b) in one segment, d2 and all of class b are missing; with one unlabelled document ahead of 1,000 documents each of classes a, b and c, all 1,000 class-c documents are missing. If the unlabelled documents come last, only they are lost. Separately, a segment with no class field at all makes split throw NullPointerException from terms(classFieldName).size() before its try, so the three writers it opened are never closed and the output directories stay locked. Every release from 6.1.0 to 10.5.2 and main is affected (the NullPointerException since 7.0.0); indexes in which every document has a value of the documented single-valued sorted doc-values class field are not. Counting one extra group, null-checking the terms and opening the writers inside the try fixes both; the repro, the whole classification test suite and the module's checks pass with that change.

• DatasetSplitter (lucene/classification, org.apache.lucene.classification.utils): "Utility class for creating training / test / cross validation indexes from the original index" (DatasetSplitter.java:44-46). split(reader, train, test, cv, analyzer, termVectors, classFieldName, fieldNames...) copies each document into one of the three outputs; the class field "must be indexed with sorted doc values" (:74-75).
• Grouping. split runs GroupingSearch on the class field over all documents, groups sorted by index order (the position of each group's first document), and copies the documents of the returned groups group by group (:119-164).
• Null group. By default the grouping puts documents without the group field into one extra group with the value null; setIgnoreDocsWithoutGroupField(true) leaves them out instead (GroupingSearch.java:63, :393-403). split does not call it.

1. split opens its three IndexWriters first (:92-94).
2. It counts the classes: per segment, the value count of the class field's sorted doc values; a segment without them also adds terms(classFieldName).size() ("approximate with no. of terms", :96-115). Documents without a value are never counted.
3. It asks for that many groups: gs.search(indexSearcher, MatchAllDocsQuery.INSTANCE, 0, noOfClasses) (:126). On one segment with at least one unlabelled document (and an exact count, step 6) there are noOfClasses + 1 groups, so the group whose first document comes last is cut, whole, from all three outputs. That group is a labelled class whenever the first unlabelled document comes before it (in [u, a, b, c] class c goes, class a stays); otherwise it is the unlabelled documents.
4. For a segment where no document has the class field, terms(classFieldName) is null ("or null if it has none", LeafReader.java:134) and :112 throws NullPointerException. That line is before the try (:117), so the finally that closes the writers and the reader (:176-179) never runs.
5. Several segments: the count adds up per-segment dictionaries, so when classes repeat across segments it exceeds the real number and the null group fits. [d0 unlabelled, d1 = a] + [d2 = b] still loses d2; [d0, a, b] + [a, b] keeps everything.
6. Deleted documents: the count is the segment's dictionary size, and a class value left only on deleted documents still counts. One segment [d0 unlabelled, d1 = a, d2 = b, d3 = c] with d3 deleted and not merged copies d0, d1 and d2 (count 3); after forceMerge(1) the count is 2 and d2 is lost again.

In the three-document case the outputs are test = [d0], cross-validation = [d1], training empty; with the fix test = [d0, d2], cross-validation = [d1].

• Who is exposed. Applications that build training, test and cross-validation indexes with DatasetSplitter.split (opt-in lucene/classification module; not marked experimental) from an index where some live documents have no class value. Nothing in Lucene calls it outside tests, and a code search of the Apache organisation found no caller in Solr.
• Unlabelled documents are a normal input to the module. Its classifiers skip them on purpose (LUCENE-5338), and its own test base indexes "unlabeled doc" and "second unlabeled doc" (ClassificationTestBase.java:227, :333). That is classifier support, not a statement about DatasetSplitter, which documents no contrary restriction.
• Default call; one segment exposes it. No setting is involved: any ratios, fieldNames null or not. TestDataSplitter, the class's only regular (non-@Monster) test, force-merges to one segment and passes a single leaf (TestDataSplitter.java:78-80); the originalIndex javadoc still reads "an {@link org.apache.lucene.index.LeafReader} on the source index" (DatasetSplitter.java:68); and from 6.1.0 to 6.3.0 split accepted only a LeafReader. How common each layout is was not studied.
• Silent. split returns normally and the outputs are well-formed indexes that just lack a class: a classifier trained on them never learns it, and its evaluation never tests it.
• Realistic data (one workload, not a prevalence study). 500 documents, 4 classes with skewed frequencies (rarest 3%), about 2% unlabelled at random positions, force-merged to one segment: 64 of 100 trials lose labelled documents (1,386 in total). The same data left in several segments: 0 of 100.
• Bounded. At most one group per split, cut whole; no deleted, duplicated or foreign document appears in any output, CheckIndex is clean and the source index is unchanged (3,000 random indexes).
• Second symptom. A segment with no class field (for example a flush of only unlabelled documents) throws NullPointerException and leaves the three output directories locked; on a file-system directory the lock is held until the JVM exits, and the caller's reader is left open (on success split closes it).
• How likely. Plausible: a little-used utility, and multi-segment indexes with repeated classes hide it; nobody has reported it since 6.1.0 (2016).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected d2 (the only document of class b) in one of the outputs, but they hold [d0, d1]
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.assertTrue(Assert.java:42)
       org.apache.lucene.classification.utils.TestDatasetSplitterUnlabelledDocs.testUnlabelledDocumentDropsLabelledClass(TestDatasetSplitterUnlabelledDocs.java:72)

• The contract. "Utility class for creating training / test / cross validation indexes from the original index" (DatasetSplitter.java:44-46); the class field "must be indexed with sorted doc values" (:74-75), a property of the field, not a rule that every document has a value; the only documented failure is "@throws IOException if any writing operation fails on any of the indexes" (:78). The contract is implied, not literal: no javadoc promises that every document lands in exactly one output.
• The grouping exists for class coverage. LUCENE-7196 (#8251, "DataSplitter should be providing class centric doc sets in all generated indexes") added it, committed as "guaranteed class coverage in split indexes through grouping by class". Before that, split copied every hit of a MatchAllDocsQuery (6.0.1), so this is a regression of that change.
• The count was meant to be enough. The code comment says "get the exact no. of existing classes" (:96); LUCENE-7591 (#8642) calls its fallback "an approximate count (upper bound)".
• A missing class field was handled before. 6.1.0 checked if (terms != null) and otherwise fell back to 10,000 groups; on main, LUCENE-7591 replaced the later IllegalStateException (the LUCENE-7400 version) with the terms fallback but no null check. The module has fixed the same null-Terms pattern elsewhere: LUCENE-7303 (#8357, "Avoid NPE if classField doesn't exist in SNBC") and LUCENE-7950 (#8999).
• Unlabelled documents are expected. LUCENE-5338 made the classifiers skip them, and the module's test base indexes them (ClassificationTestBase.java:227, :333). TestDataSplitter, the only regular splitter test, labels every document (TestDataSplitter.java:64-80), which is why it never showed (the other caller, the @Monster Test20NewsgroupsClassification, is skipped by default); no javadoc, CHANGES or MIGRATE entry mentions documents without a class value.
• Index order alone decides. Of 996 single-segment indexes of 1-5 documents that mix labelled and unlabelled documents, 660 lose a labelled document and 336 lose only unlabelled ones; all 363 fully labelled ones are correct, and the other 5 of the 1,364, with no labelled document at all, throw an exception (not with the fix).

The strongest "works as designed" case, built by the intended-lens review (another model family) and the devil's advocate: a supervised dataset is fully labelled; the class field "must be indexed with sorted doc values"; the only regular upstream test labels every document; there is no explicit promise of a complete partition; and 6.2.0-6.6.6 deliberately threw IllegalStateException for a segment without sorted doc values.

Why it fails: nothing documents a fully labelled precondition, and the module's classifiers handle unlabelled documents on purpose, with no contrary restriction in DatasetSplitter. The cut group is chosen by index order, so a labelled class can vanish while the unlabelled documents are kept, the opposite of any "ignore unlabelled" design. LUCENE-7591 replaced the old exception with a fallback meant to handle such segments, so the NullPointerException is not an intended rejection, and leaving the writers locked is wrong for any exception. Verdict: a defect (adjudication: confirmed, high; every verifier: real; the cross-model lenses rated the intended-behaviour argument weak; a later fact check by the same model family rated it moderate for the labelled-class loss and weak for the NullPointerException half, and suggested inviting maintainers to choose whether unlabelled documents are kept or left out).

Unlabelled documents keep being copied, as today when they are not the cut group. A maintainer may prefer to leave them out on purpose with setIgnoreDocsWithoutGroupField(true) (option B); either removes the labelled loss. The two options are a policy choice (keep or leave out the unlabelled documents), not two ways to the same output. Workaround until a fix ships: give every document a value of the sorted doc-values class field (for example an explicit "none" class), or delete the unlabelled documents and commit before opening the reader for split; that route was run on main for one segment and for a segment without the class field (IndexWriter drops a fully deleted segment); a merge policy that keeps fully deleted segments was not tried.

After the class count, add noOfClasses++ ("documents without a class value form one more group"); in the terms fallback, add only when terms != null; declare the three writers as null before the try and create them as its first statements (IOUtils.close in the finally skips nulls). +14/-6 in DatasetSplitter.java.

        Terms terms = leave.reader().terms(classFieldName);
        if (terms != null) {
          noOfClasses += terms.size();
        }
      }
      noOfClasses += valueCount;
    }
    // documents without a class value form one more group (the null group)
    noOfClasses++;

    IndexWriter testWriter = null;
    IndexWriter cvWriter = null;
    IndexWriter trainingWriter = null;
    try {

Trade-offs

• Unlabelled documents form their own group, so they also get the test and cross-validation ratios. The extra group changes nothing when every document is labelled.
• It also removes the empty-index topNGroups must be >= 1 (got 0) error seen in PR #12942 (the limit is then 1).
• A failure while opening a writer now reaches the caller wrapped in IOException by the existing catch, and the finally closes the caller's reader, as for every failure inside the try today (read, not run separately).
• Not changed: Terms.size() may return -1 when a codec does not store it (the default codec does); a SortedSet or terms-only class field still fails in the grouping even when every document is labelled (a separate lead).

What the checks validated

• Scratch worktree of main a9779ef3c193, gradle: the new test passes 2 of 2; the whole :lucene:classification:test passes with assertions on and off (13 suites, 45 tests, 0 failures, 1 @Monster test skipped).
• :lucene:classification:check -x test passes (checkGoogleJavaFormat, ecjLintMain, ecjLintTest, forbiddenApisMain, forbiddenApisTest).
• Standalone runner with the patched class shadowed: this doc's test (-ea, -da), the finder's (2/2) and the independent reproducer's (4/4) pass; the devil's advocate probe passes its 4 tests of this bug, including all 1,364 enumerated indexes.
• 3,000 random indexes (several segments, deletions, empty indexes): 0 losses, 0 NullPointerExceptions, 0 empty-index errors (stock main: 264 labelled losses, 151 unlabelled-only losses, 264 NullPointerExceptions, 12 empty-index errors); the 3,001-document cases copy every document.

Same as the recommended option, but call gs.setIgnoreDocsWithoutGroupField(true) (added in PR #15167) instead of counting an extra group.

Trade-offs

Fits "classifiers skip unlabelled documents", but changes the outputs: unlabelled documents are never copied, while today they usually are. The empty-index error stays (the limit is still 0; add Math.max(1, noOfClasses)). The flag is honoured by the returned groups, which is all split reads; totalGroupCount still counts the null group. Validated partly: this doc's test passes 2 of 2 with it; the finder's and the independent reproducer's tests then fail only where they expect the unlabelled document to be copied.

Pass an upper bound that cannot be short.

Trade-offs

The first grouping pass sizes its comparators and its group map by the limit (FirstPassGroupingCollector.java:116, :121), so memory grows with the index instead of with the number of classes. Not run.

Method: semi-formal (the hunt's tier for tool-guided finds that are not a formal tool's own output): a bounded-exhaustive check of the real jar, a hunt unit (classification, k-NN and utils against reference semantics), 2026-10-27, stock main jars.

• The harness enumerated every index of 1-4 documents: class none, a or b; 1 or 2 values of a stored text field; one segment or one per document; no deletion or the first document deleted; fieldNames null or {id, text, cls}; ratios 0.1/0.1 or 0.5/0.25.
• At n = 4: 24,816 splits; 7,680 lost documents (every one with a live unlabelled document; none duplicated a document or copied a deleted one; the count mixes unlabelled-only and labelled losses), 9,264 threw NullPointerException (every one with a segment without the class field), and none failed when every document had a class value.
• It was then reduced to a public-API JUnit test, and independent AI tools reproduced, reviewed and judged it (see the checks). The devil's advocate's enumeration separates the labelled losses: 660 of 996 mixed single-segment indexes.