IndexWriter buffers the first doc-values update of each field twice

IndexWriter keeps doc-values updates in a per-field buffer (FieldUpdatesBuffer) until it applies them. When an update arrives for a field that has no buffer yet, BufferedUpdates.addNumericUpdate / addBinaryUpdate create the buffer with that update, and the constructor already records it as entry 0; the methods then call addUpdate (or addNoValue) with the same update, which stores it again as entry 1. This happens whenever a field first appears in a buffered-updates packet: the writer's global buffer and each per-thread buffer.

Effects on main 5adff78bb168: the first 1 MiB binary update of a field adds 2,162,956 bytes to IndexWriter.ramBytesUsed() and each later one 1,048,576, so a writer flushes sooner than its RAM buffer setting implies; when the packet is applied the copy is resolved and applied again, so every matching document gets a second pending in-memory update (InfoStream: 1 field updates; 20 new updates for one update matching 10 documents), and for a binary value that roughly halves the size at which LFV-838 closes the writer. Final values stay correct, because the copy is identical and adjacent, and only the last pending value per document is written. A numeric field whose updates all carry one value through one term field, with no unset update (soft deletes normally qualify), skips the copy when applying, but still stores it.

Lucene 7.6.0 stored each update once; the double insert came with LUCENE-8590 (b0e6a92ede3, 7.7.0). Creating the buffer with the first update and not adding that update again fixes it.

1. Public updates reach the buffer through DocumentsWriterDeleteQueue.java:500-511: NUMERIC and SORTED_NUMERIC go to BufferedUpdates.addNumericUpdate (:507), BINARY to addBinaryUpdate (:510).
2. BufferedUpdates.java:146-156 (addNumericUpdate, :134-144, is the same) creates a missing buffer with fieldUpdates.computeIfAbsent(update.field, _ -> new FieldUpdatesBuffer(fieldUpdatesBytesUsed, update, docIDUpto)) and then always calls buffer.addUpdate(update.term, update.getValue(), docIDUpto) (:151) or buffer.addNoValue(update.term, docIDUpto) (:153); numFieldUpdates.incrementAndGet() (:155) counts the call once.
3. The constructor already records the update: FieldUpdatesBuffer.java:49 private int numUpdates = 1;, :76 appends the term, :99 stores the numeric value, :111 appends the binary value. addUpdate (:182-200 numeric, :207-212 binary, value appended at :210) and addNoValue (:202-205) call append (:214-217), which appends the term again and returns numUpdates++: the copy is entry 1.

1. RAM: the term bytes (and the value bytes, when the update has a value) are appended twice, so they count twice in ramBytesUsed().
2. Apply: FrozenBufferedUpdates logs numFieldUpdates as "field updates" (:107, :214-218) but iterates every stored entry (:246-348): for the copy it seeks the term again (:267) and adds the value again for every matching live document (:342-343, counted as "new updates"). A binary value goes into BinaryDocValuesFieldUpdates.add (BinaryDocValuesFieldUpdates.java:94) twice per document, which is what roughly halves the LFV-838 threshold. These are pending in-memory updates: when they are written, the iterator keeps only the last update per document (DocValuesFieldUpdates.java:441-448), so the doc-values file gets each document once.
3. Exception: finish() sorts the terms only for a numeric buffer with one shared value, no unset updates and one term field (FieldUpdatesBuffer.java:224, hasSingleValue :268-271); that iterator skips equal adjacent terms (:370-388), so there the copy is stored and counted but not applied again.

With the fix the first update adds 1,114,340 bytes (1 MiB plus 65,764 bytes of one-time per-field setup) and the second 1,048,616, so on main the first update costs the fixed first update plus exactly one more entry (1,114,340 + 1,048,616 = 2,162,956). Numeric fields (earlier checks on main a9779ef3c193): two numeric updates with different values, the first matching 10 documents, give 20 new updates; a single numeric update (the one-value path) gives 10.

• Who is hit: callers of updateNumericDocValue (IndexWriter.java:1962), updateBinaryDocValue (:1996), updateSortedNumericDocValue (:2037), updateDocValues (:2070) and softUpdateDocument(s) (:1938, :1664), on any configuration, deterministically: one extra copy of the first update per field per packet (the global buffer and each per-thread buffer). tryUpdateDocValue (:1721) does not go through BufferedUpdates and is not affected.
• Where it costs something: large binary values. The copy counts in the RAM buffer, so the writer flushes sooner than setRAMBufferSizeMB implies, and when applied the value is added twice per matching document to the pending in-memory updates. Through LFV-838 that roughly halves the size at which one binary update closes the writer: in the recorded run 1,100 documents x 1 MiB (1.07 GiB) closed it when the update was the field's first in the packet, and committed when another update of the field came first (assertions off, 8 GB heap; run in the original verification, not re-run for this doc).
• Where it is cheap: for numeric updates and soft deletes the copy is little more than one term's bytes (an equal value, field and docIDUpto are not stored again while all entries share them, FieldUpdatesBuffer.java:135-180, :189); soft deletes normally carry one shared value, so they are not applied twice.
• Not affected: final doc values; this bug alone throws nothing.
• Priority Low: correct results, and the cost is bounded by one extra copy per field per packet.

The update terms match no document, so the test measures only the buffering. It sits in org.apache.lucene.index to match Lucene's test layout; it uses no package-private API. The file carries the standard ASF license header and passes checkGoogleJavaFormat. It uses the 9.x test-framework package, so it does not compile against 7.x/8.x; those releases were checked with an equivalent plain-Java probe.

cp $REPRO/TestFirstDocValuesUpdateBufferedTwice.java $LUCENE/lucene/core/src/test/org/apache/lucene/index/
cd $LUCENE && ./gradlew :lucene:core:test --tests org.apache.lucene.index.TestFirstDocValuesUpdateBufferedTwice -Ptests.seed=DEADBEEF
# assertions off fails the same way: add -Ptests.asserts=false

# gradle, main 5adff78bb168, seed DEADBEEF (the same with -Ptests.asserts=false)
TestFirstDocValuesUpdateBufferedTwice > testFirstUpdateIsAccountedOnce FAILED
    java.lang.AssertionError: first=2162956 second=1048576
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:3046A8F0F574721E]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.assertTrue(Assert.java:42)
        at org.apache.lucene.index.TestFirstDocValuesUpdateBufferedTwice.testFirstUpdateIsAccountedOnce(TestFirstDocValuesUpdateBufferedTwice.java:45)
:lucene:core:test (FAILURE): 1 test, 1  failure

# standalone runner, the same message on main (-ea, -da), 9.12.3 (-da) and 10.6.0 (-da); passes with the fix shadowed

# supporting probe (not the repro), main 5adff78bb168, assertions off
ram: first=2162956 second=1048576 third=1048576
applied(precede=false): BD: applyDocValuesUpdates N msec for 1 segments, 1 field updates; 20 new updates
applied(precede=false): docs with the new value = 10 of 10
applied(precede=true): BD: applyDocValuesUpdates N msec for 1 segments, 2 field updates; 10 new updates
# with the fix: ram: first=1114340 second=1048616 third=1048576; 1 field updates; 10 new updates

• The class's invariant: the constructor records update 0 (numUpdates = 1, FieldUpdatesBuffer.java:49). TestFieldUpdatesBuffer.testBasics (TestFieldUpdatesBuffer.java:35-88), added in the same commit (as testBascis, renamed with the same body in 0650f995f9f), builds the buffer with update id:1, adds four more with addUpdate, and asserts the iterator returns exactly five entries, id:1 first and once.
• The counter disagrees with the buffer: numFieldUpdates counts one per call (BufferedUpdates.java:143, :155), and InfoStream prints 1 field updates; 20 new updates for one update matching 10 documents.
• History: 7.6.0 kept one LinkedHashMap<Term, NumericDocValuesUpdate> / <Term, BinaryDocValuesUpdate> per field, keyed by the update term (a repeated term replaced its entry), so each call was stored at most once, and it passes the RAM probe. b0e6a92ede3 (LUCENE-8590, "Optimize DocValues update datastructures", 2018-12-04; backport eaf840867fa in 7.7.0) added FieldUpdatesBuffer and the computeIfAbsent plus unconditional addUpdate pattern in one diff; no CHANGES entry, code comment or review comment on lucene-solr PR #513 mentions storing the first update twice.
• Reviewers already call the pattern a bug: in the review of the unmerged PR #16214 (in-place vector updates), iprithv wrote on 2026-06-08 (https://github.com/apache/lucene/pull/16214#discussion_r3374995773) about the PR's new addVectorUpdate: "The first update for a field gets added twice ... Harmless due to last-write-wins but wastes memory and looks like a bug". The author fixed that method; its code comment there reads "otherwise the first update would be buffered twice". The review comment also said addNumericUpdate/addBinaryUpdate "have an if/else to avoid this"; they do not: their if/else only picks addUpdate or addNoValue.

The case for intended: both classes are package-private; setRAMBufferSizeMB says "the account of RAM usage for pending deletions is only approximate" (LiveIndexWriterConfig.java:174); the copy is idempotent, so no documented result changes; and FieldUpdatesBuffer.java:50-54 says terms are stored "without de-duplication".

Why it still holds: the "approximate" note goes on to explain delete-by-Query, whose RAM Lucene cannot measure (:175-177); here a measured value is appended twice, so the bytes really are held twice. The no-de-duplication comment is about separate calls that repeat a term, not one call stored twice. And the copy is really applied, twice the per-document work, which also roughly halves the LFV-838 threshold. This supports calling the impact minor, not calling the code intended.

One option was built and validated; the other is described, not run. The fix for LFV-838 is separate (in BinaryDocValuesFieldUpdates) and does not replace this one.

In addNumericUpdate and addBinaryUpdate: look the buffer up; if it is missing, create it with the update (the constructor records it) and stop there; otherwise call addUpdate/addNoValue as now. This is the shape the #16214 author used for the vector method.

FieldUpdatesBuffer buffer = fieldUpdates.get(update.field);
if (buffer == null) {
  // the constructor records this update as the buffer's first entry: do not add it again
  fieldUpdates.put(
      update.field, new FieldUpdatesBuffer(fieldUpdatesBytesUsed, update, docIDUpto));
} else if (update.hasValue) {
  buffer.addUpdate(update.term, update.getValue(), docIDUpto);
} else {
  buffer.addNoValue(update.term, docIDUpto);
}
numFieldUpdates.incrementAndGet();

Trade-offs

• Changes only what is stored (one entry less per field per packet), not the order in which updates are applied, so results are unchanged.
• fieldUpdates is a plain HashMap (BufferedUpdates.java:66), so get + put behaves like computeIfAbsent; the extra lookup happens once per field per packet.
• The constructor already handles both hasValue cases (FieldUpdatesBuffer.java:80-83, :98-103, :110-112).

What the checks validated

• Scratch worktree of 5adff78bb168, gradle, OpenJDK 25.0.3, seed DEADBEEF. Stock: the repro fails, assertions on and off.
• With the fix: the repro plus TestFieldUpdatesBuffer, TestBufferedUpdates, TestFrozenBufferedUpdates, TestDocValuesFieldUpdates, TestBinaryDocValuesUpdates, TestNumericDocValuesUpdates, TestMixedDocValuesUpdates, TestSortedNumericDocValuesUpdates, TestIncrementalDocValuesUpdates, TestPendingSoftDeletes, TestSoftDeletesDirectoryReaderWrapper, TestSoftDeletesRetentionMergePolicy, TestAtomicUpdate and TestRollingUpdates: 155 tests, 0 failures, 2 skipped by assumptions, assertions on and off.
• :lucene:core:checkGoogleJavaFormat, ecjLintMain, ecjLintTest, forbiddenApisMain and forbiddenApisTest pass with the fix and the test.
• The probe with the fix: the first 1 MiB update adds 1,114,340 bytes; one update matching 10 documents gives 10 new updates.

Start every buffer empty and always call addUpdate/addNoValue.

Trade-offs

More invasive: the constructor uses the first update to seed fields[0], docsUpTo[0] and numericValues[0], which the one-field and one-value compaction relies on, and TestFieldUpdatesBuffer builds buffers that way, so it would need changes too.

• Arithmetic: one 1 MiB binary update matching 2,000 documents should append 2,000 MiB, below the 2,147,483,631-byte array limit, yet it overflowed at destination index 2^31, which needs 2,047 earlier 1 MiB appends: about two appends per document.
• Probe: an executed same-package probe of the buffered-updates classes on the stock main jar showed the field buffer iterating the first update twice while counting it once.
• Code and repro: the root cause was then read in BufferedUpdates and FieldUpdatesBuffer and reduced to the public-API test.
• Reviews: an independent reproduction, reviews by a model from another family (OpenAI GPT-5.6), an intended-behaviour review and a final review; see Independent checks.