IndexWriter#tryUpdateDocValue accepts a doc id outside [0, maxDoc), returns success and updates a different (or a nonexistent) document

IndexWriter#tryUpdateDocValue(reader, docID, fields) (Expert) updates doc values of one document named by its number in an NRT reader, and is also the documented way to soft-delete or un-delete one document. Its helper tryModifyDocument, shared with tryDeleteDocument, never checks that the number lies in [0, reader.maxDoc()): the leaf-reader branch uses it as is, and the composite branch only asserts. The id goes into a per-segment update buffer that packs each doc id plus a flag bit into an entry of at least bitsRequired(maxDoc - 1) + 1 bits (the packed array may round the width up to 8, 16, 32 or 64 bits) and keeps only the entry's low bits; its only guard is assert doc < maxDoc. So on a 4-document segment, id -1 is stored as doc 3: the call returns 7 (a value > 0, "the update succeeds") and doc 3 gets the new value, with assertions on or off. On other segment sizes the same call still returns success but leaves an entry for a nonexistent document (doc 15 on 10 documents, doc 127 on 40, 64 or 100). With assertions off, the production default, ids at or past maxDoc on a composite reader wrap onto other documents (8 -> doc 0, 9 -> doc 1, NO_MORE_DOCS -> doc 7), or are kept as an entry for a nonexistent document, which the commit persists, CheckIndex calls clean and forceMerge(1) shifts onto the next segment's documents. Every trigger is a caller bug, so the priority is low; but the call reports success while it changes a different, valid document (or adds an entry for a nonexistent one). It is in every release since 7.4.0 and on main; a 7-line range check at the top of tryModifyDocument fixes it and passes the whole core test suite.

// IndexWriter.java:1791-1805
private synchronized long tryModifyDocument(IndexReader readerIn, int docID, DocModifier toApply)
    throws IOException {
  final LeafReader reader;
  if (readerIn instanceof LeafReader lr) {
    // Reader is already atomic: use the incoming docID:
    reader = lr;                                       // :1796  no check at all
  } else {
    // Composite reader: lookup sub-reader and re-base docID:
    List<LeafReaderContext> leaves = readerIn.leaves();
    int subIndex = ReaderUtil.subIndex(docID, leaves); // id >= last docBase -> last leaf
    reader = leaves.get(subIndex).reader();
    docID -= leaves.get(subIndex).docBase;
    assert docID >= 0;                                 // :1803  only asserts
    assert docID < reader.maxDoc();                    // :1804
  }
  ...                                                  // :1823-1824  toApply.run(docID, rld); return seqNo

// DocValuesFieldUpdates.java
// :253  bitsPerValue = PackedInts.bitsRequired(maxDoc - 1) + SHIFT;   (SHIFT = 1, :42)
// :254  docs = new PagedMutable(1, PAGE_SIZE, bitsPerValue, PackedInts.DEFAULT);   (width may round up)
// :373  assert doc < maxDoc;                                          (-1 passes)
// :384  docs.set(size, (((long) doc) << SHIFT) | hasValueMask);       (only the entry's low bits survive)

1. The buffer. tryUpdateDocValue creates a NumericDocValuesFieldUpdates sized for the segment's maxDoc (IndexWriter.java:1740-1744) and adds the caller's leaf id unchanged (:1756-1757).
2. The width. Each entry has at least bitsRequired(maxDoc - 1) + 1 bits: PackedInts.DEFAULT lets the packed array round the width up to 8, 16, 32 or 64 bits when that costs at most 25% more (PackedInts.java:186-213, since LUCENE-8598 in 7.7.0), and Packed64.set does not mask, so only the low bits of that width survive. For maxDoc 4 the width is exactly bitsRequired(3) + 1 = 2 + 1 = 3 bits. -1 << 1 | 1 is all ones; its low 3 bits are 0b111; shifted back, that is doc 3.
3. The guards. The leaf branch has none; the buffer's assert doc < maxDoc holds for -1. So the update is applied with assertions on or off, and the call returns docWriter.getNextSequenceNumber() (IndexWriter.java:1824): 7.
4. Same arithmetic, other ids (assertions off, composite reader, where only the asserts at :1803-1804 stand in the way): maxDoc 8 gives 4 bits, so 8 -> doc 0, 9 -> doc 1 and NO_MORE_DOCS (2^31 - 1) -> doc 7. maxDoc 10 gives 5 bits, so 12 fits and is kept: an entry for a document that does not exist, written to the new doc-values generation at commit.
5. Other segment sizes (leaf reader, id -1, assertions on and off): -1 lands on the last document only when the width is exact and maxDoc is a power of two (4, 8 or 16 documents). On 10 documents it becomes an entry for nonexistent doc 15, and on 40, 64 or 100 documents for nonexistent doc 127 (8-bit entries: 7 bits rounded up for 40 and 64, exactly 8 for 100). Every one of these calls returns success.
6. After the commit the doc-12 entry is an extra FieldExistsQuery hit, and stored fields reject doc 12; CheckIndex does not flag that commit on any version tested, and forceMerge(1) shifts the next segment's values by one document.

With assertions on, the composite asserts stop the 8-doc and 10-doc rows before any change.

• Who is hit. Callers of the Expert IndexWriter#tryUpdateDocValue, including its documented uses to soft-delete and un-delete single documents (IndexWriter.java:1712-1715; CHANGES.txt:7417-7421), that pass an id outside the reader they pass. Plausible caller bugs: a -1 or NO_MORE_DOCS "not found" value, an off-by-one, an id from another or newer reader, or a top-level id together with a leaf reader (the study's probe, assertions off: two 10-document segments, the second segment's leaf with top-level id 17, meant for its doc 7, returns 26 and writes into its doc 1, because only the low 4 bits of 17 are kept).
• No special setup. Default IndexWriterConfig and codec, a 4-document segment, one thread, deterministic. The leaf-reader -1 call returns success at every segment size tried (4 to 100 documents), with assertions on or off, so Lucene-style -ea test runs do not catch it; which document it hits depends on the size. The composite cases need assertions off, the production default.
• What goes wrong. A different, valid document gets the value, or is soft-deleted and so disappears from search, while the call returns its success value; or the commit holds an entry for a document that does not exist, CheckIndex calls it clean, and the next merge shifts values across ten other documents and drops one.
• What does not go wrong. Correct callers, with ids from the same NRT reader, are unaffected. No Lucene main code calls tryUpdateDocValue; IndexRearranger (lucene/misc) calls tryDeleteDocument with ids from a loop over maxDoc. No memory corruption: the wrong data goes through ordinary Java arrays.
• Impact class: data corruption, likelihood contrived (needs a caller bug); priority low.

The test lives in package org.apache.lucene.index by Lucene test convention but uses no package-private members. The seed sets the test framework's random locale, time zone, default codec, similarity and a few other settings; the outcome does not depend on them: 10 of 10 and 20 of 20 derived seeds fail. Use this one test as the regression test too.

The same failure, by the hunt repro harness: main 645d4fe14a20, 10.5.2 and 10.6.0, assertions on and off. A framework-free main() with the stock codec prints the same result. The wider assertions-off cases (the table under "The finding in 30 seconds") come from the study's 7-test class and are supporting evidence, not part of the repro.

# gradle, main 645d4fe14a20, seed DEADBEEF, assertions on (codec=Asserting(Lucene104))
TestTryUpdateDocValueNegativeDocId > testNegativeDocIdOnLeafReader FAILED
    java.lang.AssertionError: tryUpdateDocValue(leaf, -1) returned 7; doc 3 expected:<3> but was:<777>
        at org.apache.lucene.index.TestTryUpdateDocValueNegativeDocId.testNegativeDocIdOnLeafReader(TestTryUpdateDocValueNegativeDocId.java:48)
1 test completed, 1 failed

# gradle, same, -Ptests.asserts=false: the same failure and message
# repro harness, the same message: main, 10.6.0 and 10.5.2, -ea and -da (1 run, 1 failed each)
# with the fix: the test passes (gradle, assertions on and off)

• Contract. IndexWriter.java:1707-1712: "If the provided reader is an NRT reader obtained from this writer, and its segment has not been merged away, then the update succeeds and this method returns a valid (> 0) sequence number; else, it returns -1 ...". No doc-id precondition and no "undefined" clause. CHANGES.txt:7417-7421 (LUCENE-8297): "With tryUpdateDocValues users can update doc-values fields for individual documents. This allows for instance to soft-delete individual documents."
• The asserts show such ids are not meant to get through (an inference from the code; no comment says so). The composite branch has asserted docID >= 0 and docID < reader.maxDoc() since LUCENE-4203 (2012), and the buffer doc < maxDoc since LUCENE-7868 (2017). When tryDeleteDocument was added (4.0), its delete path also asserted the range (ReadersAndLiveDocs.delete), on the leaf branch too; with assertions off, the 4.0 BitVector threw ArrayIndexOutOfBoundsException for most bad ids, though ids between maxDoc and the next multiple of 8 returned true and deleted nothing (executed on the 4.0.0 jars). LUCENE-8297 (2018) then sent the same unchecked id into a packed buffer that truncates it; its discussion (#9344) never mentions validation.
• Lucene rejects out-of-range ids elsewhere on the same reader. BaseCompositeReader.java:261-266 throws IllegalArgumentException "docID must be >= 0 and < maxDoc"; CodecReader.java:93-101 checks with Objects.checkIndex under the comment "Don't trust the codec to do proper checks"; TestDirectoryReader.testOOBDocID (:1090-1097) expects the exception. IndexWriter#addBatch checks its doc ids too (ColumnValidation.java:297-307).
• The same call rejects other bad input with IllegalArgumentException, for example a field without doc values ("can only update NUMERIC or BINARY fields! field=id", IndexWriter.java:2090-2093), executed in the tie-break probe.
• No cost argument. The method is synchronized and already does a linear search over segments (IndexWriter.java:1814-1819).

Strongest case for "intended" (caller misuse, so the fix is hardening): a doc id is by definition in [0, maxDoc) (IndexReader.java:51-52: "non-negative integers which each name a unique document"); these methods are Expert; Lucene often documents out-of-range ids as the caller's problem (Bits.get: "The result of passing negative or out of bounds values is undefined by this interface, just don't do it!", Bits.java:30-31; DocValuesIterator.advanceExact: the target "must be a valid doc ID") and guards internal invariants with asserts; in LUCENE-4203 (#5275) the issue's reporter and committer, Michael McCandless, wrote that "you should use an NRT reader from IW anyway to even find the docID to delete"; and open PR #16847 keeps the range check assert-only. All read-only cross-model reviews made this point.

Why it fails:

1. Those "undefined" clauses are stated in the javadoc of read APIs; tryUpdateDocValue's javadoc has none, and it is a write API that persists the damage.
2. The damage is not confined to the caller's bad input: a different, valid document changes, under a success return value.
3. The writer commits an internally inconsistent index (a doc-values entry beyond maxDoc) that CheckIndex accepts and a merge spreads onto other documents.
4. The leaf branch has not even an assert, so Lucene's own -ea test runs cannot catch the negative-id case.
5. The same reader, the same call and addBatch reject bad input with exceptions; the merged #16919 (2026-10-28) turned more explicit AssertionErrors into proper exceptions in other files, the same direction.

What remains is framing (bug or hardening), which sets the low priority, not the verdict. A "hardening" label is likely; the trigger is a caller error.

A is validated and recommended. Open PR #16847 (long doc ids) rewrites these lines with an assert-only toLeafDocId; the check belongs before the id is re-based. Workaround for users today: check 0 <= docID && docID < reader.maxDoc() for the exact reader you pass before calling tryUpdateDocValue or tryDeleteDocument.

Covers both branches, negative ids and both try* methods, with the message BaseCompositeReader#readerIndex already uses; constant cost.

@@ -1791,6 +1791,13 @@ public class IndexWriter
   private synchronized long tryModifyDocument(IndexReader readerIn, int docID, DocModifier toApply)
       throws IOException {
     final LeafReader reader;
+    // Validate up front, like BaseCompositeReader#readerIndex: an out-of-range id would otherwise be
+    // truncated to the segment's doc-id width by the doc-values update buffer and land on another
+    // (or a nonexistent) document.
+    if (docID < 0 || docID >= readerIn.maxDoc()) {
+      throw new IllegalArgumentException(
+          "docID must be >= 0 and < maxDoc=" + readerIn.maxDoc() + " (got docID=" + docID + ")");
+    }
     if (readerIn instanceof LeafReader lr) {

Trade-offs

Callers that pass out-of-range ids get an exception instead of silent success. For tryDeleteDocument that replaces today's "> 0 but nothing deleted" (maxDoc 10, id 10) or ArrayIndexOutOfBoundsException (maxDoc 64, id 64); a composite -1 changes from IndexOutOfBoundsException to IllegalArgumentException; an out-of-range id on a reader that is not a SegmentReader now gets the range message instead of "the reader must be a SegmentReader ...", since the check runs first. With #16847's long ids the same check works on the long.

What the checks validated

• Scratch worktree of main 645d4fe14a20 (removed afterwards), seed DEADBEEF: stock, the test fails through gradle with assertions on and off; with A it passes.
• Closest suites with A, -Ptests.asserts=true and false: the repro, TestTryDelete, TestRollingUpdates, TestIndexWriter, TestMixedDocValuesUpdates, TestNumericDocValuesUpdates, TestBinaryDocValuesUpdates, TestInfoStream, TestIndexWriterDelete, TestSoftDeletesRetentionMergePolicy (every core test that calls a try* method, plus the doc-values update suites) and lucene/misc TestIndexRearranger (the in-tree tryDeleteDocument caller): 246 tests in 11 classes, 9 skipped, 0 failures each time.
• All of :lucene:core:test with A (assertions on): 8,898 tests in 749 classes, 322 skipped, 0 failures.
• Earlier, with A shadowed in: the study's 4-test and 7-test classes pass with assertions on and off.

Replace the two composite asserts with if ... throw and add the same check on the leaf branch, or make #16847's LeafReaderContext.toLeafDocId throw instead of assert.

Trade-offs

Same result; more lines, and it must also handle ReaderUtil.subIndex returning -1 for a negative composite id. Not run.

Turn assert doc < maxDoc (DocValuesFieldUpdates.java:373) into a range check that throws.

Trade-offs

Defence in depth for any future caller, but it sits on the internal path of every doc-values update, where ids are already valid; A alone fixes this bug. Not run.

Document that the id must be in [0, reader.maxDoc()).

Trade-offs

Leaves silent success with a different document changed; not recommended.

• Method (non-formal): code reading in a hunt unit that ran controlled-concurrency testing (Fray) and stress tests of tryDeleteDocument and tryUpdateDocValue against flush, merge and commit.
• Signal. While writing that unit's oracle "returns > 0 and changes a different document", the finder noticed that tryModifyDocument only asserts the doc id. Probes on the stock main jar with assertions off showed the wrong-document and nonexistent-document effects.
• Reduction. The finder wrote a public-API JUnit (4 tests); an independent reproducer's own 7-test class found that the leaf-reader -1 case fails with assertions on too; a later minimization kept only that case, as the one test above.
• The unit's Fray and stress hunt found other registered try* bugs, not this one.