IndexWriter#addBatch: a throwing dense LongColumn cursor gives later documents another document's doc value

IndexWriter#addBatch (experimental, since 10.5.0) copies a dense LongColumn into the field's NUMERIC or SORTED_NUMERIC doc-values buffer in bulk, one 256-value page at a time, and registers the batch's documents only after the whole copy has returned. If the application's LongValuesCursor throws after at least one page was filled during the call, the batch is rejected correctly (the exception propagates, the writer stays open, the batch's documents are deleted), but the values of the filled pages stay buffered with no document. At flush the k-th document with the field is paired with the k-th buffered value, so every later document of that field in the same in-memory segment is stored with the value of a document that many positions earlier (for a SORTED_NUMERIC field with an earlier multi-valued document: the counts stay right, the values come from other documents). Nothing throws, CheckIndex is clean, and the values survive commit, reopen and forceMerge(1). Default configuration and codec; it needs an application cursor that throws, so it is rare. The sibling dense SORTED path and the stock flat writer of the new dense vector path (#16734) keep documents and values aligned; the fix is a few lines of bookkeeping.

• Columnar addBatch. IndexWriter#addBatch(ColumnBatch) (experimental, 10.5.0) indexes a batch column by column. A DENSE LongColumn has a value for every document and hands it over through a LongValuesCursor that the application writes (LongColumn.values(), document/column/LongColumn.java:79-88).
• The doc-values buffer. NumericDocValuesWriter and SortedNumericDocValuesWriter keep the values of the in-memory segment in a PackedLongValues.Builder (pages of 256 values, PackedLongValues.java:34, :66) and the documents that have the field in a docsWithField set.
• Non-aborting exception. An exception that rejects the current documents but keeps the writer and the in-memory segment for later documents.

1. IndexingChain.processDenseLongColumn runs the doc-values pass first and calls addDenseValues on the field's writer (IndexingChain.java:1076-1093).
2. addDenseValues calls pending.add(cursor) and only after it returns registers the batch's documents: docsWithField.addRange, lastDocID / currentDoc, and pendingCounts for SORTED_NUMERIC (NumericDocValuesWriter.java:81-94, SortedNumericDocValuesWriter.java:73-102).
3. PackedLongValues.Builder.add(LongValuesCursor) packs a full page, asks the cursor for at most the rest of the current page with fillDocValues, then advances pendingOff and size (PackedLongValues.java:265-279). When a later fillDocValues call throws, the values of the earlier calls stay counted.
4. DocumentsWriterPerThread.updateBatch treats the exception as non-aborting: it deletes the batch's documents and keeps the segment (DocumentsWriterPerThread.java:349-354, :399-409).
5. The NRT reader, the flush and the index sorter read the buffer through BufferedNumericDocValues, which takes the next document of docsWithField and the next buffered value (NumericDocValuesWriter.java:192-198; single-valued SORTED_NUMERIC uses it too, SortedNumericDocValuesWriter.java:192-199). With k left-over values every later document gets the value k positions earlier (up to the first k get the rejected batch's values); the last k values are never read, so nothing throws. Once an earlier document of a SORTED_NUMERIC field was multi-valued, the per-document counts stay right but the values are read from the same shifted stream (SortedNumericDocValuesWriter.java:192-199, :279-289).

Writer state from an executed probe: after documents 0-254, 255 values and 255 documents. The 2-document batch: fill call 1 (offset 255, length 1) returns -1, the page is packed, fill call 2 (offset 0, length 1) throws: 256 values, still 255 documents. Documents 257-259 (42, 43, 44) are then paired with -1, 42 and 43; the good document's 44, the last buffered value, is never read. Other executed cases: a 257-document SORTED_NUMERIC batch failing on its last value leaves 256 values (the next document reads -1 instead of 42); a 600-document batch failing at value 520 leaves 512 (42, 43, 44 read -1, -2, -3). Not affected: a throw inside the first fill call, the sparse (tuple) long path, the dictionary SORTED dense path, and a flush right after the failure.

• Who is hit. Users of the experimental addBatch / updateBatch with a DENSE LongColumn for NUMERIC or SORTED_NUMERIC doc values (including a LongField-style schema with points plus SORTED_NUMERIC; the doc-values pass runs first), whose LongValuesCursor throws a RuntimeException after the batch crossed a 256-value page boundary of the field's buffer, and who keep adding documents with that field before the segment flushes. updateBatch keeps the writer and the in-memory segment for that, and Lucene's own recovery test indexes more documents afterwards (after a commit()); the public addBatch contract says nothing about unchecked exceptions.
• Default or opt-in. Default IndexWriterConfig and codec, no assertions needed; reproduced with NoMergePolicy, the default TieredMergePolicy and the randomized test configuration. The opt-in part is the experimental API and a cursor that throws: Lucene ships no LongValuesCursor outside tests, so the cursor is application code, for example one that decodes or validates its source lazily and fails on a bad row.
• Scale. 255 earlier documents and a 2-document batch are enough. The shift equals the number of values in the completed fill calls: 1, 256 and 512 in the executed cases.
• The damage. Documents whose own add calls succeeded carry another document's value: sorting, doc-values queries, facets and aggregations on that field, and an index sort (probe: the good document reads -1 with the sort on the field itself or on another field). Points stay correct, so points and doc values of one field disagree; LongField.newExactQuery can return wrong hits when IndexOrDocValuesQuery runs on doc values. CheckIndex is clean; the values survive commit, reopen and forceMerge(1). Repair means finding and reindexing the affected documents; for a doc-values-only field that is not an index-sort field, updateNumericDocValue / updateSortedNumericDocValue can also rewrite them (IndexWriter.java:1962, :2037), but not for a LongField-style field with points.
• Exposure today. The API shipped in 10.5.0 (June 2026). The only downstream user found, Elasticsearch's batch-indexing path, is unreleased, behind a feature flag and a setting that defaults to off, with no known throw site here.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<42> but was:<-1>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:647)
       org.junit.Assert.assertEquals(Assert.java:633)
       org.apache.lucene.index.TestDenseLongColumnFailure.testFailedDenseBatchShiftsLaterValues(TestDenseLongColumnFailure.java:86)

• Design note (DocumentsWriter.java:76-79): "All other exceptions ("non-aborting exceptions") can still partially update the index structures. These updates are consistent, but, they represent only a part of the document seen up until the exception was hit. When this happens, we immediately mark the document as deleted so that the document is always atomically ("all or none") added to the index." updateBatch follows it (DocumentsWriterPerThread.java:349-354). addDocument: "if an Exception is hit ... then the index will be consistent" (IndexWriter.java:1465). The addBatch javadoc (:1528-1536), the column javadocs, CHANGES and MIGRATE never ask for a flush after a failed batch.
• Lucene's tests deliberately throw from a dense cursor and check the writer afterwards. ColumnBatchTestUtil has throwing-cursor helpers "so tests can exercise the failure-cleanup path in DocumentsWriterPerThread#updateBatch" (:589-591), including ThrowingDenseLongColumn (:638-642). TestColumnBatchIndexing#testLongValuesCursorThrowMidBatchRecovers (:740-792) asserts 777, 111 and 222 for the documents indexed afterwards. That shows intent to keep the writer usable, not a written contract for same-segment recovery. It misses the bug because its 6 values throw inside the first fill call and because it commits right after the failure (:760-762, "so the cleanup-induced fully-deleted segment is pruned"). With a 300-value batch failing at 280 the test passes with that commit() and fails without it: expected:<777> but was:<1>.
• Two sibling paths do it right. SortedDocValuesWriter.addDenseOrdinalValues (:156-182, #16101) registers exactly the processed documents in a finally; a Z3 model that counts values and documents found no batch of up to 4096 values where the counts differ after a throw. The dense vector path merged in #16734 writes the rule down: "The cursor may throw while it is being consumed ... the writer must remain in a consistent state: every doc ID it has recorded must have its vector ... record doc IDs and vectors only after fill has returned" (KnnFieldVectorsWriter.java:46-50), and its new test keeps the failed and the recovery batch in one segment on purpose (TestColumnBatchIndexing.java:886-922). That holds for the stock flat writer; LFV-579, a separate defect, shows HNSW and scalar-quantized wrappers over a custom flat writer that fall out of step.
• History. #16270 made batch exception handling match updateDocuments. #16553, backported to 10.5.2, is a precedent: a non-aborting exception left indexing-chain state inconsistent (a stored-fields frame left open), so that a later flush failed; reviewers called it a corruption issue and pulled it into the 10.5.2 release candidate.

The strongest case for "works as designed" (independent checkers, both still concluding "real"): the API, LongColumn and LongValuesCursor are experimental; the cursor "produces exactly size() values" (LongValuesCursor.java:22-23), so a cursor that throws within range is a broken producer; PackedLongValues.Builder promises no atomicity; addBatch declares only IOException; and the upstream test flushes before indexing again, as if only post-flush recovery were intended.

Why it fails: the documents that end up wrong are later ones whose own add calls succeeded, and no contract lets a successful add store another document's value. The same javadoc requires implementations to throw on overrun (:25-26, :57-58), so exceptions belong to the cursor protocol (that javadoc covers overrun, not an in-range failure, so the contract stays ambiguous), and Lucene's own helper throws in range on purpose. updateBatch keeps the segment and nothing says to flush; the test's commit() is there, by its comment, to prune the deleted segment. A stale comment in testDenseColumnCountMismatchThrows ("avoid the partially-written DV entries", TestColumnBatchIndexing.java:291-292) dates from a PR revision that wrote values and documents together, so it is about values on deleted documents, not misalignment. Verdict: a defect (a correctness bug of an experimental API; a maintainer may still label it an enhancement).

One fix was built and validated through gradle; the others are design alternatives that were not built.

In both addDenseValues methods, remember pending.size() before the bulk add and, in a finally, register as many documents (and, for SORTED_NUMERIC, pendingCounts entries) as values were added, like SortedDocValuesWriter.addDenseOrdinalValues. The registered doc ids are the batch's own documents, which updateBatch deletes right after, so no live document gains a value. +34/-23 lines in two files; the snippet is the NumericDocValuesWriter half.

final long sizeBefore = pending.size();
try {
  pending.add(cursor);
} finally {
  // Register a doc for every value that reached the buffer, even if the cursor threw part-way,
  // so that values stay aligned with docsWithField.
  final int added = Math.toIntExact(pending.size() - sizeBefore);
  if (added > 0) {
    docsWithField.addRange(firstDocID, firstDocID + added);
    lastDocID = firstDocID + added - 1;
  }
  updateBytesUsed();
}

Trade-offs

• The rejected batch's deleted documents keep values until a merge drops them, as with the per-document path; nothing reads them through live docs.
• Cost: constant bookkeeping per non-empty dense batch (two size() reads, a subtraction, a Math.toIntExact and a branch); no added work per value on the success path and no extra copy.
• It relies on Builder.size() counting exactly the completed fill calls, which add(LongValuesCursor) does by construction (PackedLongValues.java:274-276).
• The finally also runs updateBytesUsed() after a failure; the stock code skips it, so the in-memory segment's RAM accounting misses the kept values until the writer's next successful update.

What the checks validated

• stock, the test, seed DEADBEEF, assertions on: 1 classes, 3 tests, 3 failures, 0 errors, 0 skipped (the 3 tests fail).
• stock, the test, assertions off: 1 classes, 3 tests, 3 failures, 0 errors, 0 skipped.
• stock, :lucene:core:check -x test: BUILD SUCCESSFUL.
• fix, the test, assertions on: 1 classes, 3 tests, 0 failures, 0 errors, 0 skipped.
• fix, the test, assertions off: 1 classes, 3 tests, 0 failures, 0 errors, 0 skipped.
• fix, the test, seed CAFEBABE, tests.iters=10: 1 classes, 30 tests, 0 failures, 0 errors, 0 skipped.
• fix, closest suites, assertions on: 21 classes, 909 tests, 0 failures, 0 errors, 62 skipped.
• fix, closest suites, assertions off: 21 classes, 909 tests, 0 failures, 0 errors, 62 skipped.
• fix, :lucene:core:check -x test: BUILD SUCCESSFUL.
• fix, all of :lucene:core:test, seed DEADBEEF: 749 classes, 8,892 tests, 0 failures, 0 errors, 322 skipped.
• Repro harness, fix shadowed on main a9779ef3c193: the finder repro 2 of 2 pass, the paste-ready test 3 of 3, the index-sort probe 2 of 2.
• Earlier: the finder's fix check, the independent reproduction's 6 tests (judge re-run), the cross-model reviewers' tests with 77 and 72 relevant existing tests, the security-probe tests and the finder repro at the introducing commit all pass with the fix.
• The cross-model review's own persistence test, run by this doc's fact check on main a9779ef3c193 (assertions on): stock 1 of 2 fail (expected:<7> but was:<1000>), with the fix shadowed 2 of 2 pass.

On a throw, restore size, pendingOff and the packed pages.

Trade-offs

Not built. Harder than it looks: the loop may already have packed a page that also holds earlier documents' values, so a rollback has to undo a pack or keep a copy.

Copy the whole batch out of the cursor before touching the buffer.

Trade-offs

Not built. Atomic and simple, but every value is copied once more, which gives up the point of the bulk path.

Discard the in-memory segment.

Trade-offs

Not recommended: it drops every document since the last flush, against updateBatch's non-aborting handling (#16270) and the recovery Lucene's own test checks.

Method: SMT (Z3) model of the dense doc-values page loop, then failure injection on the real jar.

• The hunt unit covered the columnar addBatch path. Reading the dense writers while modelling them raised the hypothesis; the model (dense_commit.py) encodes PackedLongValues.Builder.add(LongValuesCursor) and the writers' bookkeeping. Property W1, "dense long DV page loop failure-atomic", is violated (sat) with the smallest witness 255 values already in the page and a 2-document batch failing on its 2nd value (1 value kept, 0 documents); no violation for batches that fit in the current page's free slots (unsat). Property W3 found no violation for the dictionary SORTED dense path (batches up to 4096 values). The model is bounded (W1: up to 4 pages, 1,024 values) and counts values and documents only, not buffer contents or RAM accounting.
• A failure-injection sweep on the stock jar then threw from the cursor at every point of a grid: 301 of 605 points violated for dense NUMERIC and for dense SORTED_NUMERIC, with assertions on and off; 0 of 605 for the sparse long paths.
• The finder wrote the 2-test JUnit repro and a fix; verifiers then reproduced it with their own tests, and an adjudication re-ran their tests and the finder's with new seeds (Independent checks).