addIndexes(CodecReader...) counts deleted documents against the doc limit since 9.9.0, rejecting inputs whose live documents fit

IndexWriter.addIndexes(CodecReader...) copies only the live documents of its inputs and checks the doc limit (MAX_DOCS, 2,147,483,519) against live documents only. Since 9.9.0, when no index sort is configured, addIndexesReaderMerge first wraps all inputs of the merge in a SlowCompositeCodecReaderWrapper so that OneMerge#reorder can renumber documents. The wrapper builds a MultiReader, which counts every doc ID, deleted ones included, and throws IllegalArgumentException: Too many documents: composite IndexReaders cannot exceed ... when they add up to more than the limit. The default reorder() returns null, so the view is built and thrown away.

So a call whose inputs fit by live documents but not by maxDoc() fails cleanly and adds nothing: at full scale with no test hook, two indexes of 1,073,741,824 doc IDs with 1,024 live documents each fail in one call on main and 10.5.2, succeed in two calls, and succeed in one call on 9.8.0. It needs more than 2^31 doc IDs across the inputs of one call, so it is rare. Introduced by #12622 (9.9.0); every release from 9.9.0 to 10.6.0 and main is affected. Building the view only when the inputs' maxDoc() sum fits fixes it; that fix passes the reduced-scale repro, the closest core suites, TestBPReorderingMergePolicy and core lint (not run at full scale), but it silently skips a custom reorder() for such an oversized merge.

1. Admission counts live documents. addIndexes(CodecReader...) (IndexWriter.java:3275) sums leaf.numDocs() (:3294) and calls testReserveDocs(numDocs) (:3296). The default MergePolicy.findMerges(CodecReader...) puts all readers into one OneMerge (MergePolicy.java:627-631), whose totalMaxDoc is the sum of numDocs() (MergePolicy.java:245-254); after success addIndexes reserves that sum (IndexWriter.java:3344, :3356). addIndexesReaderMerge checks live documents once more (:3504, :3518).
2. The eager view. With no index sort and no input that has blocks without a parent field (:3548-3556), addIndexesReaderMerge runs CodecReader mergedReader = SlowCompositeCodecReaderWrapper.wrap(readers); (:3561) and only then merge.reorder(mergedReader, directory, intraMergeExecutor) (:3562). If reorder returns null, the input readers are merged as they are (after OneMerge#wrapForMerge, :3542-3546, which by default returns each reader unchanged).
3. The view counts deleted documents. wrap returns a single reader as is (SlowCompositeCodecReaderWrapper.java:54); for two or more it builds the wrapper, which adds up reader.maxDoc() into int doc starts (:73) and creates new MultiReader(...) over the inputs (:95). BaseCompositeReader sums maxDoc() as a long (BaseCompositeReader.java:86) and above IndexWriter.getActualMaxDocs() (:89) throws IllegalArgumentException("Too many documents: composite IndexReaders cannot exceed ...") for a reader that is not a DirectoryReader (:99-107).
4. The view was not needed. The default OneMerge#reorder returns null (MergePolicy.java:325-328; javadoc "will be called when index sorting is disabled on a merged view of the OneMerge", :311-324). SegmentMerger (IndexWriter.java:3569) gets those input readers and drops deleted documents: without the view, 9.8.0 writes the merged segment at full scale, and the proposed fix passes the reduced-scale tests (it was not run at full scale).
5. How it fails. The exception is thrown before SegmentMerger exists, so nothing is written; with the default ConcurrentMergeScheduler the merge fails on the merge thread, which also reports it as an uncaught MergeException, and addIndexes rethrows the merge's exception (:3382); with SerialMergeScheduler (the minimal test) it comes straight out of handleMergeException (:5056). Either way only an Error is treated as tragic (:3390). Each failed call leaves its merge in the writer's merge-exception list (:5040-5041, :5819-5824) until the next forceMerge clears it (:2250) or the writer is closed: a probe counted 20 retained entries after 20 failing calls.
6. Who reaches it. One index's own maxDoc is capped at MAX_DOCS (a DirectoryReader above it is rejected as corrupt, BaseCompositeReader.java:90-98), and a single input never builds a MultiReader, so at real scale one merge needs two or more input readers whose maxDoc() sum exceeds the limit: for example the leaves of two indexes (the executed runs), or the leaves of one index passed twice, as TestIndexWriterMaxDocs.testAddTooManyIndexesCodecReader repeats one reader (read, not run). The ordinary merge path builds the same view (IndexWriter.java:5557), but there the writer has reserved every input document, so its maxDoc sum stays within the limit.

• Who is hit. Callers of addIndexes(CodecReader...) with the default configuration (no index sort) that pass two or more inputs (for example the segments of two indexes) holding more than 2,147,483,519 doc IDs in total, deleted or filtered-out ones included, while their live documents fit: for example heavily deleted, or FilterCodecReader-filtered, views of two indexes of about a billion documents each.
• What goes wrong. The call is refused with a misleading message (the result would hold a fraction of the limit) and adds nothing. It fails cleanly: in a probe of 20 failing calls in a row (assertions on and off) no file was written, no document reservation leaked, the inputs were untouched, the writer stayed open and not tragic, and CheckIndex was clean afterwards. Each failed call does leave one entry in the writer's merge-exception list until the next forceMerge or close.
• Workarounds. Split the inputs so that each merge's maxDoc() sum fits: one addIndexes call per input index (executed at full scale), or a findMerges(CodecReader...) override that splits them into groups (executed at small scale). An index sort on the destination is not a general workaround: it skips the view, but a sorted destination accepts only inputs that already have a congruent index sort and rejects unsorted ones with cannot change index sort (IndexWriter.java:3245-3251).
• Likelihood: rare. Few deployments combine more than 2^31 doc IDs in one call, and the recorded searches found no public report since 9.9.0 (2023).

Public API plus the test framework's setIndexWriterMaxDocs, ByteBuffersDirectory, SerialMergeScheduler so the merge runs on the test thread; deterministic, single thread. The limit is lowered only after the source index is built. At real scale one index cannot exceed the limit, so one merge needs two or more readers whose maxDoc() sum does, such as the leaves of two indexes; the full-scale runs with no test hook (in the observed output) show the same failure.

With the default ConcurrentMergeScheduler the test fails the same way, and the merge thread's uncaught MergeException is reported as well. As the regression test it can be used as is: it has the license header and passes core lint.

TestAddIndexesDeletedDocsLimit > testOnlyLiveDocsCountTowardsLimit FAILED
    java.lang.AssertionError: expected: 2 live docs added under a limit of 2; actual: Too many documents: composite IndexReaders cannot exceed 2 but readers have total maxDoc=3
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:827B29374A95FE3C]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.apache.lucene.index.TestAddIndexesDeletedDocsLimit.testOnlyLiveDocsCountTowardsLimit(TestAddIndexesDeletedDocsLimit.java:48)

:lucene:core:test (FAILURE): 1 test, 1  failure

# the same with -Ptests.asserts=false. Where it is thrown (default ConcurrentMergeScheduler, main 645d4fe14a20):
org.apache.lucene.index.MergePolicy$MergeException: java.lang.IllegalArgumentException: Too many documents: composite IndexReaders cannot exceed 2 but readers have total maxDoc=3
    at __randomizedtesting.SeedInfo.seed([DEADBEEF]:0)
    at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:761)
    at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:753)
Caused by: java.lang.IllegalArgumentException: Too many documents: composite IndexReaders cannot exceed 2 but readers have total maxDoc=3
    at org.apache.lucene.index.BaseCompositeReader.<init>(BaseCompositeReader.java:104)
    at org.apache.lucene.index.MultiReader.<init>(MultiReader.java:76)
    at org.apache.lucene.index.MultiReader.<init>(MultiReader.java:50)
    at org.apache.lucene.index.SlowCompositeCodecReaderWrapper.<init>(SlowCompositeCodecReaderWrapper.java:95)
    at org.apache.lucene.index.SlowCompositeCodecReaderWrapper.wrap(SlowCompositeCodecReaderWrapper.java:56)
    at org.apache.lucene.index.IndexWriter.addIndexesReaderMerge(IndexWriter.java:3561)
    at org.apache.lucene.index.IndexWriter$AddIndexesMergeSource.merge(IndexWriter.java:3463)
    at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:661)
    at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:726)

# full scale, no test hook, main ea57360912ed (same four files), assertions off:
inputs: leaves=2048 sum maxDoc=2147483648 (MAX_DOCS+129) sum numDocs=2048
CONTROL two calls: OK numDocs=2048
ONE CALL: FAILED java.lang.IllegalArgumentException: Too many documents: composite IndexReaders cannot exceed 2147483519 but readers have total maxDoc=2147483648
# 10.5.2: the same; 9.8.0: ONE CALL: OK numDocs=2048

• Live documents everywhere. The up-front check (IndexWriter.java:3294-3296), OneMerge(CodecReader...)'s totalMaxDoc (MergePolicy.java:245-254), the merge's own check (:3504, :3518) and the final reservation (:3344, :3356) all count live documents. The javadoc's only doc-limit failure is "if addIndexes would cause the index to exceed MAX_DOCS" (:3272-3273), and MAX_DOCS is the limit on documents "added to the index" (:201-203). The view is the one place that counts deleted documents.
• Unused by default. The default reorder() returns null and the view is discarded; the same inputs succeed in two calls or with one merge per reader. For inputs that are already index-sorted, the destination's index sort alone decides: two sorted inputs fail into an unsorted destination and succeed into one with the same sort, which skips the view (executed at small scale). Unsorted inputs cannot take that route, since a sorted destination rejects them.
• A regression. 9.8.0 accepts the identical full-scale call. #12622, which added the view, never discusses doc limits, addIndexes or MAX_DOCS in its 45 comments and reviews; its author describes the view as "the reordering logic that works on a merged view of the input segments and doesn't care about deletes". No CHANGES.txt, MIGRATE.md, javadoc or test entry announces a limit on the inputs' maxDoc() sum.
• Deleted and filtered inputs are normal here. TestAddIndexes.testFakeAllDeleted adds an all-deleted filtered reader; MultiPassIndexSplitter ("the input data comes from the input index with artificially applied deletes") and PKIndexSplitter pass filtered leaves. TestIndexWriterMaxDocs tests the addIndexes limit only with all-live inputs.

The strongest case for "works as designed", rated moderate by the independent reproducer and cross-model reviews (root-cause and intended lenses); the refute lens rated it weak and the intended-behaviour devil's advocate found it does not survive: a composite reader has one int doc-ID space, OneMerge#reorder is documented to receive a merged view whenever index sorting is off, so a merge whose inputs cannot be addressed as one reader is arguably unrepresentable; findMerges(CodecReader...) (whose javadoc mentions one merge per reader) or separate calls are the documented knobs. It also needs more than about 2.1 billion input doc IDs and fails loudly before anything is written, with an easy workaround, so maintainers may see a surprising limitation rather than a contract violation.

Why it fails: with the default configuration nothing uses the view, every document-limit check counts live documents, for already index-sorted inputs the destination's index sort alone decides whether the limit applies, and the call worked in 9.8.0 with no announcement of the change. Where it does hold: a merge policy that really overrides reorder(), such as BPReorderingMergePolicy, needs the view, so a fix must decide what that policy gets for an oversized group. Maintainers may also prefer a javadoc change, since the @throws clause does not literally say "only if".

One fix was built and validated on 645d4fe14a20 with gradle; the alternatives come from the adjudication and the checks and were not run.

In addIndexesReaderMerge, add the inputs' maxDoc() sum to the condition that builds the view. When it does not fit, the merge runs on the input readers, exactly as without reordering, and drops deleted documents. A custom reorder() is then silently not called for that group (see the trade-offs).

// addIndexes only reserves live docs, so the inputs' maxDoc (deleted docs included) may add up
// to more than a single (composite) reader can address: then do not offer a merged view
final long totalMaxDoc = readers.stream().mapToLong(CodecReader::maxDoc).sum();
if (hasIndexSort == false
    && hasBlocksButNoParentField == false
    && readers.isEmpty() == false
    && totalMaxDoc <= getActualMaxDocs()) {
  CodecReader mergedReader = SlowCompositeCodecReaderWrapper.wrap(readers);
  ...

Trade-offs

• For an oversized group a merge policy that overrides reorder() is not called, so that group is merged without reordering, silently (a probe counted reorderCalls=0). BPReorderingMergePolicy already treats reordering as best-effort: it skips it on NotEnoughRAMException and records reordered=false in the segment diagnostics when reorder() was not called. An infoStream message for the skipped case would make it visible (not run).
• Cost: one pass over the inputs' maxDoc() per merge.

What the checks validated

• Scratch worktree of 645d4fe14a20, OpenJDK 25.0.3, seed DEADBEEF: stock, the minimal test fails with assertions on and off; with the fix it passes, on and off.
• With the fix: TestAddIndexes 39 of 39, TestIndexWriterMaxDocs (19 run, 2 skipped), TestIndexWriterMergePolicy (18 run, 3 skipped), TestOneMergeWrappingMergePolicy and TestGetVectorCountCodecWrappers pass: 83 tests, 5 skipped, 0 failures.
• With the fix: TestBPReorderingMergePolicy passes (1 run, 2 nightly tests skipped; an earlier check ran the nightly tests with the fix: 3 of 3 pass).
• Core lint (google-java-format, ecj, forbidden APIs) passes with the fix and the test in place.
• Not run at full scale: every fix run lowered the limit with the test hook.

For example a OneMerge method that says whether it reorders, or a view built lazily; give a clear error only when a reordering policy meets an oversized group (the adjudication's preference).

Trade-offs

Keeps the default configuration free of the view and makes the oversized case explicit, but needs a small API change or an override check; not run.

Group the readers so that each OneMerge's maxDoc() sum fits.

Trade-offs

Reordering still runs per group, but one call then produces several segments, unlike the documented default of one, and custom findMerges overrides still hit the limit; not run.

State that, without an index sort, the inputs of one merge must fit MAX_DOCS by maxDoc(), deleted documents included.

Trade-offs

No code change, but keeps the 9.9.0 regression and makes the limit depend on index sorting; not run.

Method: semi-formal (an SMT query encoding a hypothesis formed while modelling, then executed tests).

• In a hunt unit (CodecReader wrappers) the finder modelled SlowCompositeCodecReaderWrapper's doc-ID arithmetic in Z3 (quantifier-free bit-vectors, exact Java int semantics).
• Query Q5a, with the ordinary merge path's precondition (the inputs' maxDoc() sum within MAX_DOCS, which the writer guarantees for its own segments), showed that the view's doc starts never wrap (unsat).
• Query Q5b encoded addIndexes(CodecReader...)'s actual precondition (live documents within MAX_DOCS) and asked for a maxDoc() sum above it: sat, with a concrete witness. So the view's MultiReader check is reachable from a legal call.
• The finder bridged that to the stock jar with a JUnit at reduced scale and a full-scale run with no test hook; independent checks reproduced it (see the checks below).
• Recorded as semi-formal rather than formal: the query encodes a hypothesis and its sat answer is plain arithmetic; the exception path itself was found by reading the code and confirmed by execution.