tryUpdateDocValue loses to an earlier doc-values update that is still buffered

Update a document's doc value by term with updateDocValues, then update the same document again with tryUpdateDocValue before the writer flushes or refreshes: the earlier update wins. In the repro the term update sets n=2 (sequence number 5), the try call sets n=3 and succeeds (sequence number 6), and a fresh NRT reader and the commit both show n=2. Lucene raises no exception and CheckIndex is clean. It needs only default settings and two documents, and every release since the API appeared in 7.4.0 has it; with a soft-deletes field the same mechanism loses an un-delete.

Lucene has two ways to change a document's doc value without re-indexing it:

• updateDocValues(Term, Field...) changes every document that matches a term. It is buffered: the writer keeps it in memory and resolves it later, for example at the next flush, NRT reopen or commit, when any segment is flushed, or when the buffer uses too much RAM.
• tryUpdateDocValue(IndexReader, int docID, Field...) (an "Expert" method) changes one document, addressed by its doc ID in an NRT reader of the same writer. It looks the document up right away and queues the change on that segment's pending updates, tagged with a delete generation taken at call time (see below). It returns a sequence number (or -1, meaning "retry").

Every change returns a sequence number. The IndexWriter javadoc says it "expresses the effective order in which each change was applied". So when two changes touch the same document, the one with the larger number should win.

What you get: n=2, in the fresh reader and in the commit. The later update is gone, silently. If you call flush() between steps 3 and 4, you get n=3.

With a soft-deletes field, step 3 soft-deletes id:1 by term and step 4 un-deletes it with tryUpdateDocValue(r, 0, soft=null), the way its javadoc suggests. You expect 2 live documents and get 1.

Inside the writer, conflicting doc-values updates are not ordered by sequence number but by a second counter, the delete generation. The two methods read that counter at different times:

1. Step 3 waits without a generation. updateDocValues only appends the update to the writer's global buffer (IndexWriter.java:2074 -> DocumentsWriter.java:149-151 -> DocumentsWriterDeleteQueue.java:135-139). It gets sequence number 5, but no generation yet.
2. Step 4 takes the next generation at once. tryUpdateDocValue runs long nextGen = bufferedUpdatesStream.getNextGen(); (IndexWriter.java:1729) and adds the update to the segment's pending updates (rld.addDVUpdate(updates), :1775). In the repro that is generation 2. Nothing in tryModifyDocument (:1791-1829) first gives the buffered update a generation.
3. Step 5 gives the buffered update the generation after that. When the buffer is frozen and published (here by the new NRT reader or the commit; any flush, in any thread, or a RAM-triggered apply does it too), packet.setDelGen(nextGen++) (BufferedUpdatesStream.java:75) gives it generation 3. It still applies to the old segment (FrozenBufferedUpdates.java:191).
4. The larger generation wins. Pending updates of one document are merged "favoring the largest delGen" (DocValuesFieldUpdates.java:153-168, used at ReadersAndUpdates.java:427). Generation 3 (n=2) beats generation 2 (n=3). The same merge feeds the soft-deletes count (ReadersAndUpdates.java:429), so the un-delete loses the same way.

Sequence numbers say 5 then 6; generations say 3 after 2; the generation decides. A flush between the two calls gives the term update its generation first, which is why the outcome depends on timing. Fix option C (below) shows that the order of the generations alone decides: it gives the buffered update its generation before the try call without applying it first, and the later update then wins.

• Who is hit. Applications that call tryUpdateDocValue on a document that also got a term-based doc-values update (updateDocValues, updateNumericDocValue, updateBinaryDocValue, or a soft delete by term) after the NRT reader used for the try call was opened, with nothing in between that froze the buffer (a flush, reopen, commit, a segment flush in any thread, or a RAM-triggered apply). updateSortedNumericDocValue and the sorted-numeric branch of tryUpdateDocValue (both added by #16581) take the same path (code reading, not run). Two natural patterns: a bulk update by term followed by a per-document override, and a fast path that falls back to updateDocValues(Term) when tryUpdateDocValue returns -1, as the stress test does. That fallback is legal, though on -1 the javadoc itself says to resolve the document again and retry; it recommends updateDocValues(Term) only for documents indexed after the reader was opened, which alone cannot hit this window.
• Default settings. The numeric case needs only the default IndexWriterConfig, codec and merge policy; tryUpdateDocValue is a public "Expert" API. The un-delete case needs the opt-in soft-deletes field.
• No scale needed; frequent in one concurrent workload. Two documents and a handful of calls reproduce it, with the same failure on all 5 seeds tried (they ran with different randomized test codecs). In a 4-thread, update-heavy stress test where each thread owns its documents and mixes term updates and try updates 50/50 (200 documents, 2,000 updates), 21 to 67 of the 200 documents end with an older value in each of 10 runs on main (5 with assertions on, 5 off), and 45 on 10.5.2; over all 14 stock-main runs the range is 21-68. With term updates only: 0. Other workloads were not measured.
• Silent and durable. Lucene raises no exception and no internal assertion fires under -ea; CheckIndex is clean; the wrong value is committed. A lost un-delete leaves the document soft-deleted, so a merge can reclaim it unless a retention policy keeps it.
• Looks intermittent. Anything that freezes the buffer between the two calls hides it: a flush, an NRT reopen, a commit, a segment flush that a document-count or RAM limit triggers in another thread, or a RAM-triggered apply.
• Reach. A code search found no tryUpdateDocValue callers in Elasticsearch, OpenSearch or Solr, so the known exposure is custom applications using the expert API (the search says nothing about private code).
• Security. No direct security impact identified: the trigger is the application's own correct API calls. The downstream impact depends on how the fields are used; where a doc value or a soft delete drives access control, tenant visibility or retention, a lost update would affect that too.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<3> but was:<2>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:647)
       org.junit.Assert.assertEquals(Assert.java:633)
       org.apache.lucene.index.TestTryUpdateDocValueAfterBufferedUpdate.testLaterTryUpdateDocValueWins(TestTryUpdateDocValueAfterBufferedUpdate.java:46)

• The ordering contract. IndexWriter class javadoc (IndexWriter.java:117-118): each change returns a sequence number "which expresses the effective order in which each change was applied". CHANGES.txt:8925-8927 (LUCENE-7302, which introduced them): the numbers indicate "the effective equivalent single-threaded execution order". The repro is single-threaded.
• The method's own contract is met. The tryUpdateDocValue javadoc (IndexWriter.java:1708-1720) promises that for an NRT reader of this writer whose segment was not merged away "the update succeeds and this method returns a valid (> 0) sequence number"; its only other limit is that it "can only update documents visible to the currently open NRT reader". Both hold: it returns 6 and doc 0 is visible. It also says: "This can be used to un-delete a soft-deleted document" (and CHANGES.txt:7390-7393, LUCENE-8298: "This allows to undelete a soft-deleted document").
• Lucene's own test assumes last-write-wins. TestMixedDocValuesUpdates.testTryUpdateMultiThreaded (:499) mixes updateDocValues (:531) and tryUpdateDocValue through its helper doUpdate (:534) on the same documents, records the last value written to each (:539) and asserts it at the end (:578). It passes only because doUpdate (:589-599) opens a fresh NRT reader, which freezes the buffer, right before every try call; TestSoftDeletesRetentionMergePolicy.doUpdate (:781-792) does the same. So no upstream test reaches the window.
• History. The API arrived with LUCENE-8297 (#9344, by @s1monw, reviewed by @mikemccand; 2018) taking its generation with getNextGen() at call time from its first commit; the thread does not discuss ordering. #8355 (LUCENE-7301, 2016) fixed an older doc-values update winning over a newer one inside one batch as data loss. #8398 (LUCENE-7344, open) is the same family for delete-by-query.

Strongest case for "intended": tryUpdateDocValue is an Expert, point-in-time API, and the NRT reader it needs comes from DirectoryReader.open(IndexWriter), which is marked @lucene.experimental. It updates the document as reader r sees it, and a term update buffered after r was opened is "newer than the reader", so resolving it later is consistent with r. The javadoc promises success and visibility, not last-write-wins against pending term updates, and Lucene's own test helpers open a fresh reader before every try call.

Why it fails: the call returns a larger sequence number than the term update, and sequence numbers are defined as the effective order, with no ambiguity in one thread. The visibility note is about which documents can be addressed, and this one is visible; the javadoc does not ask for a fresh reader before each call. The same two calls give n=3 when any flush happens in between, so the result follows hidden buffer state, not the reader's snapshot. Lucene's own multi-threaded test asserts last-write-wins across the two APIs. An independent reviewer from another model family (OpenAI GPT-5.6) built a similar argument and rated it weak: "Expert" and "experimental" are stability labels, "not permission for undocumented ordering reversal". Verdict: not intended.

The un-delete case, separately: IndexWriterConfig.java:557-559 says "Currently there is no API support to un-delete a soft-deleted document." That contradicts the method javadoc and the 7.4.0 CHANGES entry, so the un-delete case carries a documentation conflict of its own; it does not affect the numeric case.

The root cause is that a try update takes a generation before earlier buffered updates have one. A fix either gives the earlier updates their generation first, or declines the try path while they have none. Two options were built and validated on a scratch worktree at ea57360912ed (removed afterwards); the others are untested ideas. Every option changes only tryUpdateDocValue; tryDeleteDocument is unchanged, and whether it has an analogous ordering issue was not examined.

Before taking its generation, tryUpdateDocValue freezes the global buffer into a flush ticket, as the RAM-triggered apply does (DocumentsWriter.java:168-186), and force-publishes the ticket queue outside the IndexWriter lock (as the onDeletesApplied event does, IndexWriter.java:458-466). Under the lock, if anything is still unpublished (a segment flush still running ahead in the queue, a full flush, or a new update from another thread) it returns -1, which the javadoc already tells callers to handle. +40/-1 lines in IndexWriter.java and DocumentsWriter.java.

public long tryUpdateDocValue(IndexReader readerIn, int docID, Field... fields) throws IOException {
  if (docWriter.anyUnpublishedDeletesOrUpdates()) {
    docWriter.freezeGlobalBuffer();   // like DocumentsWriter#applyAllDeletes
    publishFlushedSegments(true);     // gives the frozen packet its delete generation
  }
  return tryUpdateDocValueInternal(readerIn, docID, fields);  // synchronized; returns -1 if
}                                                            // anything is still unpublished

// DocumentsWriter
boolean anyUnpublishedDeletesOrUpdates() {
  return anyDeletions() || ticketQueue.hasTickets() || pendingChangesInCurrentFullFlush;
}

Trade-offs

• Single-threaded callers: no extra segments. In a cost probe (1,000 rounds of add + term update + try, NoMergePolicy) all 1,000 try calls succeed and 2 segments remain, as on stock. The timed loop took 0.83-0.87 s against 0.65-1.09 s on stock (3 runs each; too few to rank them). The timer stops before the final NRT open that applies the queued packets, so the end-to-end cost, the memory of the extra small packets and the next-refresh latency were not measured.
• Concurrent writers: the call declines much more often: 73-94% of try calls returned -1 in 17 runs of the 4-thread stress test (stock: 0%). The fallback keeps the data right, but the try fast path is mostly lost under heavy concurrent updates. The check also fires on unrelated buffered hard deletes, delete-by-query and flushes, so mixed workloads may decline more (not measured).
• The public method becomes an unsynchronized wrapper around a synchronized one, because the forced publish must run without the IndexWriter lock; it can wait on the ticket-queue purge lock, and code that synchronizes on the writer no longer excludes the publish step. The publish also runs before the fields and the reader's segment are validated, so a call that then throws or returns -1 may already have published. The new -1 cause should be documented.
• Not tested: a concurrent update of the same document by another thread, issued between the check and the sequence number, can still be ordered by generation. Callers that serialize per document are not affected.
• Does not fix the sister bug LFV-010 (later term ops ignoring a try un-delete).

What the checks validated

• The repro passes 3/3 with assertions on and off; the stress test has 0 wrong documents in 5 runs with -ea and 5 with -da.
• :lucene:core:test on 17 targeted suites (the two new tests, TestMixedDocValuesUpdates, TestNumericDocValuesUpdates, TestBinaryDocValuesUpdates, TestSortedNumericDocValuesUpdates, TestIncrementalDocValuesUpdates, TestDocValuesFieldUpdates, TestSoftDeletesRetentionMergePolicy, TestSoftDeletesDirectoryReaderWrapper, TestPendingSoftDeletes, TestTryDelete, TestRollingUpdates, TestIndexWriterDelete, TestIndexWriterReader, TestIndexWriter, TestStressNRT): 321 tests, 0 failures.
• --tests 'org.apache.lucene.index.*': 216 classes, 1,398 tests, 0 failures; the new tests and TestMixedDocValuesUpdates also pass with -Ptests.asserts=false -Ptests.iters=5.
• :lucene:core:check -x test passes (google-java-format, ecjLint, forbiddenApis) after tidy re-wrapped a few lines of the new tests.

When anything is unpublished, call flush(false, false) (a full flush that neither applies deletes nor merges) before the synchronized part. The same idea as a patch experiment by the study's Alloy verifier. +21/-1 lines.

Trade-offs

• Measured cost: the cost probe makes 1,001 segments instead of 2 and takes about 25 s instead of about 0.5-0.6 s (3 runs each): every try call that follows any buffered delete or update, even of another document, runs a full flush, which writes a new segment whenever documents are buffered (the probe adds one every round).
• No new -1 results, but callers can see the change: the extra full flush creates segments, takes time, can surface flush exceptions and changes writer state. Like C, it moves synchronized off the public method and flushes before validating the fields and the reader. Same untested concurrent-race caveat as C.

What the checks validated

• The repro passes 3/3 with -ea and -da; the stress test has 0 wrong documents in 10 runs, with every try call succeeding.
• 16 targeted suites (as for C, without the stress test): 320 tests, 0 failures; org.apache.lucene.index.*: 215 classes, 1,397 tests, 0 failures.

Return -1 whenever anything is unpublished, without publishing first. Smallest patch.

Trade-offs

In single-threaded code the call would decline after every buffered update until something freezes and publishes the buffer (a new NRT reader, a flush, a commit, a segment flush or a RAM-triggered apply) (inferred from C's check; not run).

Route the doc-ID-addressed update through the delete queue so its generation follows queue order, as the study's TLA+ verifier suggested.

Trade-offs

Would also close the concurrent race, but it is a larger change to the update machinery.

State that try* updates can be overridden by earlier buffered updates until the next refresh.

Trade-offs

Contradicts the sequence-number contract; the cheapest change.

Method: model-based randomized testing (the study's non-formal baseline), with blind re-finds by two formal tools.

• A 2026-10-22 study of Lucene by AI tools covered the IndexWriter/NRT doc-values layer because an earlier bug there (now #16900) was the study's starting point. So the area was hinted; this bug was not.
• An AI tool wrote a model-based testing harness: an executable reference model of the writer, random operation sequences, and a comparison of every reader against the model. Its model-value comparison flagged this bug in a smoke run, about 4 s in.
• Alloy (SAT4J solver) and TLA+ (the TLC model checker) re-found it later. Later P, Dafny, JBMC and OpenJML checks were told about it.
• Independent AI verifiers reproduced it and voted "new bug"; this hunt re-verified it on ea57360912ed, ran the release history and the prior-art search, had it reviewed adversarially, and validated two fixes.