DatasetSplitter puts at most about half of each class into the test index, whatever testRatio is

DatasetSplitter(testRatio, crossValidationRatio).split(...) copies an index into test, cross-validation and training indexes, class by class. Both ratios are documented in the same words, as the share of the original index for that output, "between 0.0 and 1.0". But a document may enter the test index only when a running counter b is even (if (b % 2 == 0 && tc < testSize)), and the cross-validation branch has no such condition. So each class gives the test index at most about half of its documents, whatever testRatio asks for, and the documents it should have taken silently go to training (cross-validation still gets just its own share unless the two ratios together ask for more than the whole class). One class of 10 documents with testRatio 0.8 gives test/cv/training 5/0/5 instead of 8/0/2; crossValidationRatio 0.8 gives 0/8/2 as documented. It is deterministic, the same with assertions on and off, raises no error, and is in every release that has the class (4.2.0 through 10.5.2) and main. testRatio up to 0.5 is honoured up to per-class rounding. Keeping the interleave but sending every remaining document to test once it is needed to reach the test size fixes it (+6/-2 lines) and leaves splits with testRatio up to 0.5 unchanged; the repro and the whole classification test suite pass with it.

• DatasetSplitter (lucene/classification, org.apache.lucene.classification.utils): "Utility class for creating training / test / cross validation indexes from the original index" (DatasetSplitter.java:44-46). The constructor takes testRatio and crossValidationRatio, both documented as "the ratio of the original index to be used for the test IDX" ("c.v. IDX") "as a double between 0.0 and 1.0" (:55-58); it checks neither (:60-63). What is neither test nor cross-validation goes to training.
• split(...) groups the live documents by the class field (sorted doc values, :74-75), groups and documents in index order (:119-126), and sends each document to one of the three new indexes.

1. One counter, int b = 0; (:136), counts every document of every class and is never reset (b++, :162).
2. For each class, testSize = totalHits * testRatio and cvSize = totalHits * crossValidationRatio (:142-146).
3. A document goes to test only if (b % 2 == 0 && tc < testSize) (:153), else to cross-validation if (cvc < cvSize) (:156), else to training (:159-160). The cross-validation branch has no parity condition.
4. So a class of n documents gives the test index at most ceil(n/2) documents (floor(n/2) when it starts on an odd b). Below one half, tc < testSize usually stops first and the size is right up to rounding (close to one half, a class that starts on an odd b can get one test document fewer); above one half, the parity check stops first. The odd positions fall through to cross-validation until cvSize is reached, then to training, so with ratios that sum to at most 1 the missing test documents end up in training.

With testRatio 0.8 the test index holds d0, d2, d4, d6 and d8, and training d1, d3, d5, d7 and d9. On 1,000 documents in 10 random classes, shaped like Lucene's own TestDataSplitter fixture, every testRatio from 0.5 to 1.0 gives 500 test documents, and testRatio 0.7 with cross-validation 0.1 gives 500/106/394 instead of about 700/100/200.

• Who is exposed. Applications that build training, test and cross-validation indexes with DatasetSplitter (opt-in lucene/classification module; the class is not marked experimental) and pass a testRatio above 0.5, which the javadoc allows, for example a learning-curve or small-training-set experiment (70% test, 10% cross-validation, 20% training). Nothing in Lucene calls the class outside tests; the in-tree callers use testRatio 0.1, 0.2 and 0.2.
• Default vs opt-in. There is no default ratio: the caller passes both. testRatio up to 0.5 is unaffected (up to per-class rounding). No index-size condition: 10 documents are enough.
• Silent and deterministic. The same wrong split every time, with assertions on and off, and no error or log line. In a probe on a fully labelled index the outputs were well-formed (CheckIndex clean), every live document landed in exactly one of them, and the source index was untouched, so nothing looks wrong. (With unlabelled documents a labelled class can vanish from all outputs: LFV-683, a separate bug.)
• Scale. At most about half of the index can ever be test data (per class ceil(n/2) or floor(n/2), depending on where the shared counter starts); the shortfall grows with the ratio (testRatio 1.0: about half of every class is missing from the test index).
• Bounded. No crash and no corruption in the probe, and no cost that grows with the ratio: a grouped search over the live documents (a first and a second pass) and one copy loop, 8-115 ms per split whatever the ratio (main only, one synthetic 400-document index, one host, so the timing is directional).
• How likely. Rare: test sets larger than training sets are unusual, and no public report was found in the sources searched, although the class has shipped since 4.2.0 (2013).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: test index size for testRatio 0.8 expected:<8> but was:<5>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:647)
       org.apache.lucene.classification.utils.TestDatasetSplitterTestRatio.testTestRatioAboveHalf(TestDatasetSplitterTestRatio.java:49)

• The contract. "@param testRatio the ratio of the original index to be used for the test IDX as a double between 0.0 and 1.0", and "@param crossValidationRatio the ratio of the original index to be used for the c.v. IDX as a double between 0.0 and 1.0" (DatasetSplitter.java:55-58, wording unchanged since 2012). Only the cross-validation ratio is honoured over that range. No javadoc, package doc, CHANGES or MIGRATE entry mentions a cap, and the constructor checks nothing (:60-63).
• The author's stated intent. LUCENE-7196 (#8251, 2016), opened by the class's author and the change that introduced today's per-class loop: "the generated index should contain split-ratio * | docs in c | documents for each class c".
• Lucene's own test oracle. TestDataSplitter.assertSplit expects (int) (originalIndex.maxDoc() * testRatio) test documents within 20 (TestDataSplitter.java:128) but runs only testRatio 0.1 and 0.2 (:91, :95); the 2012 test asserted the exact size. On the same fixture shape, testRatio 0.7 fails that oracle: expected:<700.0> but was:<500.0>.
• No other reading explains it. The cap does not protect the training share: nothing limits cross-validation, so training can get far less than half (testRatio 0.2 with cross-validation 0.7 leaves 90 of 1,000 documents, 9%, for training).

The strongest "works as designed" case (the intended-lens review by another model family rated it moderate; the devil's advocate built it in full): the every-other-document interleave is longstanding and appears intentional, in the first version (2012) and kept by the same author through the LUCENE-7196 rewrite, although no comment, issue or commit message says why; every in-tree caller uses test ratios of 0.2 or less; test sets larger than training sets are unusual; and "between 0.0 and 1.0" could be read as the type's range, not a promise for every value.

Why it fails: keeping the interleave suggests intent about which positions are sampled, not about silently cutting the size, and the fix keeps it. The identically documented cross-validation ratio is honoured, the author's stated intent and the project's own size oracle both say size = ratio x class size, and nothing documents or enforces a limit. Verdict: a defect (adjudication: confirmed, high; every verifier: real). The cross-model fact check of this doc rates the "works as designed" case moderate and suggests the most neutral framing: a documentation and validation defect with an apparent behavioural mismatch.

A maintainer may instead prefer to document the limit or reject testRatio above 0.5 (option C), which would make this a documentation or validation defect. Workaround until a fix ships: for a test share above one half, pass it as crossValidationRatio (honoured over the whole range) and use the cross-validation output as the test index. If a real cross-validation set is needed too, swap both: pass the wanted cross-validation share as testRatio and read the test output as the cross-validation set; that share is then capped the same way, so keep it at or below 0.5 (and the two shares at most 1 together). Separate hardening note, not this bug: the constructor accepts out-of-range ratios (NaN, negative, above 1) without validation; a probe saw bounded, exception-free behaviour.

In the per-class loop of DatasetSplitter.split, count the documents seen in the class and also admit a document to the test index when the documents left in the class are no more than the test documents still missing. +6/-2 in DatasetSplitter.java.

        int i = 0;
        for (ScoreDoc scoreDoc : group.scoreDocs()) {
          ...
          // add it to one of the IDXs; interleave test documents, but take every remaining
          // document once the rest of the group is needed to reach the test size
          boolean needed = totalHits - i <= testSize - tc;
          i++;
          if ((b % 2 == 0 || needed) && tc < testSize) {

Trade-offs

• testRatio up to 0.5 gives exactly the same split as today, document by document: a transcription of the loop found no difference in the per-class counts over 92,400 configurations, nor (in the fact check) in any document's destination over 2,325,600 configurations; the fixture sweep agrees for 0.1, 0.3 and 0.5.
• Above 0.5 each class gets at least floor(n x testRatio) test documents and cross-validation at least min(floor(n x cvRatio), n - test) (same transcription, for ratios in [0, 1] that sum to at most 1, which the constructor does not enforce). So the fix rounds down above 0.5 where the stock loop rounds up below it: on the 1,000-document fixture 595, 695, 795 and 894 rather than about 600, 700, 800 and 900. The cross-validation and training documents then come from the front of each class (10 documents, testRatio 0.8: training gets d1 and d3).
• Constant extra work per document (an increment, two subtractions and one comparison); the split stays linear.

What the checks validated

• Scratch worktree of main a9779ef3c193, gradle: the new test passes 5 of 5 with TestDataSplitter; the whole :lucene:classification:test passes with assertions on and off (13 suites, 48 tests, 0 failures, 1 @Monster test skipped).
• :lucene:classification:check -x test passes (checkGoogleJavaFormat, ecjLintMain, ecjLintTest, forbiddenApisMain, forbiddenApisTest).
• Standalone runner with the fix shadowed: this doc's test (-ea, -da), the devil's advocate's upstream-oracle test (2 of 2; 0.7 with cross-validation 0.1 gives 695/106/199), the finder's (1 of 1), the independent reproducer's (7 of 7) and the 42 upstream classification tests pass.

Change if (b % 2 == 0 && tc < testSize) to if (tc < testSize): each class gives its first test-size documents to test, the next to cross-validation, the rest to training.

Trade-offs

Simplest, one line. It changes which documents are picked at every ratio, including the common 0.1-0.2 (a block instead of every other document), so existing splits are not reproducible across the change.

What the checks validated

Standalone runner only: this doc's test 5 of 5 and TestDataSplitter 2 of 2 pass; the independent reproducer's 7 tests pass.

A javadoc note on the constructor, or an IllegalArgumentException for testRatio > 0.5; the same change could validate both ratios (in [0, 1], sum at most 1).

Trade-offs

No split changes; the limit becomes explicit instead of silent. Users who want a larger test share must swap the roles of the ratios. Not run.

For example, take position i when floor((i + 1) x testRatio) > floor(i x testRatio).

Trade-offs

Honours any ratio and samples the whole class, but changes every split; more than a bug fix needs. Not run.

Method: semi-formal (the hunt's tier for tool-guided finds that are not a formal tool's own output): a bounded check of the split-size rule on the real jar, a hunt unit (classification, k-NN and utils against reference semantics), 2026-10-27, stock main jars.

• The signal came from code reading while modelling: the test branch has a parity condition the cross-validation branch lacks.
• The probe (SplitterRatio.java) split one class of 100 documents with seven ratio pairs: testRatio 0.5, 0.6, 0.8 and 1.0 all gave 50 test documents; testRatio 0 with cross-validation 0.8 gave 80; 0.1/0.1 gave 10/10 and 0.3/0.6 gave 30/60.
• It was reduced to a public-API JUnit test, and independent AI tools reproduced, reviewed and judged it (see the checks).