IndexWriter.updateBinaryDocValue keeps the caller's BytesRef, so documents still buffered in RAM get bytes written after the call

IndexWriter.updateBinaryDocValue(Term, String, BytesRef), and updateDocValues(Term, Field...) with a BinaryDocValuesField, store the caller's BytesRef object in the queued update without copying it. Documents in flushed segments get the bytes as of the call, because the writer's global buffer copies them inside the call. Documents still buffered in RAM get the bytes that BytesRef holds when the per-thread writer holding them next applies the queued updates: at the end of its next indexing operation, or when it is flushed (commit, refresh). So a caller that reuses its buffer after the call, or passes a reader's BinaryDocValues.binaryValue() straight through, silently commits the wrong value on buffered documents: a loop over one BytesRefBuilder gives every buffered target the last value. Default config and codec, one thread (every run), deterministic, CheckIndex clean; every release since 4.8.0 (2014). Binary doc-values updates are a rarely used API.

• Binary doc-values update: IndexWriter.updateBinaryDocValue(term, field, value) replaces the BinaryDocValues value of every document that matches term, without reindexing the document. updateDocValues(term, fields...) does the same for one or more Fields.
• BytesRef: a view of a byte array (bytes, offset, length). Lucene reuses BytesRef objects a lot: BytesRefBuilder.get() returns the same object after every copyChars, and a doc-values iterator refills one object for every document.
• Flushed vs buffered: a document that IndexWriter has written to a segment is flushed. A document added since the last flush (commit, refresh, or RAM-triggered flush) is buffered in the RAM buffer of a per-thread writer.

1. updateBinaryDocValue passes the caller's object on: new BinaryDocValuesUpdate(term, field, value) (IndexWriter.java:2013). updateDocValues(Term, Field...) does the same with f.binaryValue() (:2113).
2. The update stores the reference: this.value = value; (DocValuesUpdate.java:114).
3. The update is queued in the writer's delete queue (DocumentsWriterDeleteQueue.java:135-139). During the call only the global buffer reads it (tryApplyGlobalSlice, :198-215, when its tryLock succeeds; in every executed run the flushed documents got the call-time value, as that gives): that copies the bytes (FieldUpdatesBuffer.java:210, byteValues.append(value), or the constructor at :107-112 for a field's first update in that buffer), and the global buffer is what is applied to flushed segments.
4. The per-thread writer that holds the buffered documents reads the queued update only at the end of its next indexing operation (DocumentsWriterPerThread.java:361-390) or at flush (:427-439). Until then the queue holds the caller's BytesRef, so the copy for buffered documents takes whatever bytes it holds then. At flush the global packet (b) is published before the new segment, which gets a later delete generation, so it does not overwrite the new segment's own copy (IndexWriter.java:2969-2990, :6783).

One update with b to a flushed and a buffered document, then the caller writes c: flushed=b, buffered=c. A BytesRefBuilder loop over three buffered documents: new2, new2, new2. A pass-through of another index's binaryValue(): src2, src2, src2. updateDocValues with one reused BinaryDocValuesField: x2, x2. A fresh BytesRef per call: correct. If the writer indexes one more document before the caller changes the bytes, the value is correct too; that document closes the window.

• Who is hit. Callers of updateBinaryDocValue, or of updateDocValues with a BinaryDocValuesField, that change or reuse the BytesRef after the call: one BytesRefBuilder for a loop of updates, a reused byte[] wrapped in a fresh BytesRef per call (it fails the same way), or a reader-owned BytesRef such as BinaryDocValues.binaryValue() passed straight through. Callers whose BytesRef points at bytes they never change afterwards, such as a fresh BytesRef over a fresh array per call as Lucene's own tests do, never see it.
• Which documents. Targets added since the last flush or refresh that are still in a per-thread writer's RAM buffer, for updates made since that per-thread writer's last indexing operation. Which documents those are depends on flush timing (RAM buffer, refreshes), so the same code can store right values for some documents and wrong ones for others.
• Default, not opt-in. An unmodified new IndexWriterConfig() and the default codec give the same wrong values for the changed-after-return, loop and pass-through cases (on a ByteBuffersDirectory and an FSDirectory), one thread; deterministic, the same with assertions on and off. The JUnit itself sets NoMergePolicy and turns off flushing by document count. No scale: 1 to 3 documents.
• Silent and persistent. No exception; the wrong value is committed, CheckIndex reports the index clean wherever it was run (the changed-after-return, loop and pass-through cases), and NRT readers see it too.
• Reach. Binary doc-values updates are rarely used: no Lucene module calls updateBinaryDocValue outside tests, and a read-only code search of three common downstream search servers found it only in one test (code search can miss call sites).
• Age. Every release since 4.8.0 (2014). Until 7.6.0 even flushed documents got the late bytes.
• Not claimed (code reading only, not executed). With several indexing threads, each per-thread writer applies the update at its own next indexing operation; the global slice can miss its tryLock, in which case flushed segments could read late bytes on 7.7.0 and later too; and a binary field passed to softUpdateDocument(s) goes into the same kept update, which other per-thread writers apply later.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: expected:<new[0]> but was:<new[1]>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.index.TestBinaryDocValuesUpdateReusedBytesRef.testReusedBytesRefBuilder(TestBinaryDocValuesUpdateReusedBytesRef.java:50)

• The contract. "Updates a document's BinaryDocValues for field to the given value", @param value "new value for the field" (IndexWriter.java:1981-1995); updateDocValues: "Updates documents' DocValues fields to the given values" (:2060). Neither says the BytesRef is kept after the call.
• Flushing must not change the result. The IndexWriter class javadoc: "flushing just moves the internal buffered state in IndexWriter into the index" (IndexWriter.java:131-132). Here the stored value depends on whether the target had been flushed.
• The writer copies the other BytesRefs it takes during the call. addDocument consumes a BinaryDocValuesField's bytes before returning (a checker's control: one field over a builder changed between calls gives each document its own value); tryUpdateDocValue copies into BinaryDocValuesFieldUpdates (IndexWriter.java:1759-1761, BinaryDocValuesFieldUpdates.java:94-98); Term(String, BytesRef): "The provided BytesRef is copied when it is non null" (Term.java:50-55). The update's own code calls the value "a final value" (DocValuesUpdate.java:119).
• Precedent. Until 2015 Term(String, BytesRef) carried "WARNING: the provided BytesRef is not copied, but used directly". LUCENE-6821 ("TermQuery's constructors should clone the incoming term", #7879, commit 9534904c250) made Term copy, because term queries built from shared BytesRef objects are "a bit trappy"; mikemccand: "I consider this a bug fix". That trap was even documented; this one is not.
• Lucene copies reader-owned values before keeping them. The default bulk implementation in the BinaryDocValues javadoc stores BytesRef.deepCopyOf(binaryValue()) (BinaryDocValues.java:43-50). Here the object that keeps the value is IndexWriter's internal update.
• Tests and history. Upstream tests allocate a fresh BytesRef per call (TestBinaryDocValuesUpdates.toBytes, :69-83); no test asserts late-read behaviour. LUCENE-5513 (#6576), which added the API, never discusses copying.

The case for "intended". Lucene is zero-copy by habit. Field's constructor and setBytesValue say "NOTE: the provided BytesRef is not copied so be sure not to change it until you're done with this field" (Field.java:190-191, 332-333), and in LUCENE-6321 (#7382, won't fix, 2015) rmuir called modifying term bytes after construction "an abuse case". So: do not change a BytesRef you handed to Lucene.

Three more texts a maintainer may cite. The BytesRefBuilder.get() javadoc: "Any update to the content of this builder might invalidate the provided ref and vice-versa" (BytesRefBuilder.java:151-154), against the loop. The 4.9.0 CHANGES entry of LUCENE-5703: "BinaryDocValues API changed to work like TermsEnum and not allocate/copy bytes on each access, you are responsible for cloning if you want to keep data around" (lucene/CHANGES.txt:12171-12173), against the pass-through. And shaie on LUCENE-5189 ("Numeric DocValues Updates", 2013-11-21): "The updates are buffered in their raw form in memory until a flush is needed (e.g. commit(), or NRT-open)."

Why it fails. updateBinaryDocValue(Term, String, BytesRef) takes no Field, and neither it nor BinaryDocValuesField's constructor carries such a note. The NOTE does not say when the writer is "done"; for addDocument it is done when the call returns. The pass-through case changes nothing the caller owns. No reading of the NOTE explains why the stored value depends on whether the target was flushed. The same LUCENE-6321 comment goes on: "But maybe its a good idea to do a little "extra" clone work to make these core queries defensive." The builder javadoc says the ref follows the builder, not how long IndexWriter reads it, and Term(String, BytesRefBuilder) copies (Term.java:63-66). In the pass-through the caller keeps nothing: IndexWriter keeps the reader's object out of the caller's sight, while the BinaryDocValues javadoc's own example clones before keeping. The 2013 comment, from when only numeric updates existed, contrasts updates buffered "raw" (term and value) with their resolved per-document form; it does not say the caller's objects are kept.

Where it partly survives. The reused-Field case (one BinaryDocValuesField whose BytesRef the caller changes between two updateDocValues calls) is close to what the NOTE warns about, so it is the secondary case. The documented reuse idiom, setBytesValue(new BytesRef(...)) per call, is not affected. OpenAI GPT-5.6 reviews rated the "intended" argument moderate and still judged the bug real, and the GPT-5.6 fact check of this doc rated it moderate for the direct API (weaker for the pass-through) and strong for the reused-Field case; the devil's-advocate checker and the adjudication found that it fails for the direct API.

Option A was validated in a scratch worktree of main a9779ef3c193 through gradle; options B and C are design alternatives that were not run.

One line in DocValuesUpdate.java: the package-private constructor that both IndexWriter entry points call (IndexWriter.java:2013, :2113) deep-copies a non-null value. prepareForApply keeps using the private constructor, so there is no second copy. This is the remedy upstream chose for Term(String, BytesRef) in LUCENE-6821.

BinaryDocValuesUpdate(Term term, String field, BytesRef value) {
  // Copy the caller's bytes, like Term(String, BytesRef) does: documents buffered in RAM get
  // this update only after the IndexWriter call has returned.
  this(term, field, value == null ? null : BytesRef.deepCopyOf(value), BufferedUpdates.MAX_INT);
}

Trade-offs

• Cost: one array copy per binary update value (an updateDocValues call with several binary fields makes one per field), next to the copy FieldUpdatesBuffer already makes for every buffer the update is applied to.
• Reach: the constructor is also used, through buildDocValuesUpdate, by tryUpdateDocValue (IndexWriter.java:1724), which copies the bytes inside the call anyway (:1759-1761), so there the copy is extra but harmless; and by softUpdateDocument(s) (:1674, :1947) with a binary field, where it is needed: the calling per-thread writer applies the update inside the call, but other per-thread writers apply the queued node later (code reading, not executed).
• null: the null branch is reached: updateDocValues(term, new BinaryDocValuesField(field, null)) resets a value (TestMixedDocValuesUpdates.testResetValue, :628), and that test passes with the fix.
• RAM accounting: unchanged. DocValuesUpdate.sizeInBytes() and valueSizeInBytes() have had no caller since LUCENE-8590 (b0e6a92ede32) removed the last one.

What the checks validated

• The doc's test: 5/5 pass with assertions on and off (gradle, scratch worktree of main a9779ef3c193, seed DEADBEEF); stock main fails 4 of 5.
• TestBinaryDocValuesUpdates, TestNumericDocValuesUpdates, TestSortedNumericDocValuesUpdates, TestMixedDocValuesUpdates, TestIncrementalDocValuesUpdates and the new test (116 tests with two other methods that match the filter), plus TestDVUpdateBackwardsCompatibility in backward-codecs (12): 0 failures, 3 skipped by assumptions, assertions on and off.
• The whole org.apache.lucene.index test package with assertions on: 1,400 tests, 0 failures (56 skipped).
• checkGoogleJavaFormat, ecjLintMain, ecjLintTest, forbiddenApis and renderJavadoc for core pass.
• Earlier, the same code change shadowed ahead of main's jars made every checker's own test pass: the finder's 3, the reviewer's 8, the reproducer's 6, the works-as-designed reviewer's 4, the devil's advocate's 5 and (run by the fact check) the audit's 4.

BytesRef.deepCopyOf at IndexWriter.java:2013 and, for the updateDocValues path, at :2113.

Trade-offs

Same effect; keeps BinaryDocValuesUpdate a plain holder, at the price of two sites to keep in sync. It saves no copy over option A: a copy at :2113 runs inside buildDocValuesUpdate, which tryUpdateDocValue and softUpdateDocument(s) share. Not run.

Document on updateBinaryDocValue, updateDocValues and BinaryDocValuesField that the BytesRef must not change after the call.

Trade-offs

No cost, but the honest window (until the per-thread writer holding the target documents next indexes a document or is flushed) is something a caller cannot see or control, so in practice the note says "never reuse it". It keeps the flushed-versus-buffered inconsistency and the reader pass-through trap. Not run.

• The lead. The hunt plan for the hunt unit on IndexWriter update-value ownership and doc-values buffers listed it from reading main: a BinaryDocValuesUpdate keeps the caller's BytesRef, and a per-thread writer copies the bytes only at its next document or at flush. Not executed at that point.
• The witness. The finder (an AI tool) wrote a differential "mutate after return" JUnit: one update applied to a flushed and to a buffered document, then the caller changes its own BytesRef. It then classified every IndexWriter entry point that takes a caller-owned mutable object, by executed probes except for the ColumnBatch path and the Query objects of query deletes (code reading). Besides the binary update value, the probes found only the varargs arrays of deleteDocuments kept after the call (a sibling candidate); the delete queue also keeps Query objects, which were not probed; the other probed arguments are copied or consumed during the call.
• Not formal. No model, proof or exhaustive check produced the signal: the unit's bounded-exhaustive harness uses a fresh BytesRef per call and cannot see it.
• Then an independent reproducer, reviews by an independent reviewer from another model family (OpenAI GPT-5.6), a devil's-advocate checker, an adjudication and an audit re-ran it with their own tests (Checks below).