One buffered document makes IndexWriter keep every later delete and doc-values update on the heap until OutOfMemoryError

Every in-memory segment of an IndexWriter (a DocumentsWriterPerThread, DWPT) keeps a private DeleteSlice: a pointer into the writer's shared delete queue, a singly linked list of every delete and doc-values update. The pointer moves only when the segment indexes another document or is flushed. After one addDocument, a stream of only deleteDocuments(Term) or doc-values updates therefore keeps every later queue node, with its Term or update, strongly reachable.

The RAM buffer never frees them. A pure delete reaches the flush policy with no per-thread writer, so FlushByRamOrCountsPolicy can only push the global delete buffer, which frees the writer's own copy but not the chain behind the idle segment's pointer, and IndexWriter.ramBytesUsed() counts only that global buffer (reset at every push) plus the segments' own bytes. With the default config and 16-character ids the heap grows about 127 bytes per delete until OutOfMemoryError (256 MB heap: 42 MiB at 250,000 deletes, 224 MiB at 1.75 million) while ramBytesUsed() stays at 5.6-14.2 MiB; with no buffered document the same stream stays at 10-22 MiB. updateNumericDocValue and updateSortedNumericDocValue behave the same way.

Keeping these deletes until the segment flushes is required (its document must see them); the defect is that the kept span is unbounded, uncounted and never released by the RAM buffer. It came with per-thread writers in 4.0.0-ALPHA (3.6.2 flushes the document with the deletes) and is in every release up to 10.6.0 and main. commit(), an NRT reopen or flush() releases it.

• DWPT: DocumentsWriterPerThread, one in-memory segment. numRamDocs() counts the documents buffered in all of them.
• Delete queue: DocumentsWriterDeleteQueue, a singly linked list of nodes, each holding a Term[], Query[] or DocValuesUpdate[]. The class javadoc: "the garbage collector takes care of pruning the list for us. All nodes in the list that are still relevant should be either directly or indirectly referenced by one of the DWPT's private DeleteSlice or by the global BufferedUpdates slice" (DocumentsWriterDeleteQueue.java:42-45 at e1faeba08bde).
• Delete slice: DeleteSlice (DocumentsWriterDeleteQueue.java:333-372), a head and a tail pointer into the queue. Each DWPT has one (DocumentsWriterPerThread.java:134, :165); the global buffer has another.
• Push: DocumentsWriter.applyAllDeletes freezes the global buffer into a packet that is applied to the on-disk segments, and clears it (DocumentsWriter.java:168-186, DocumentsWriterDeleteQueue.java:262-277).

1. After a document is added, finishDocuments moves the DWPT's slice so that sliceHead is the queue's tail at that moment: the queue node before the first later delete (in the diagram probe, the sentinel) (DocumentsWriterPerThread.java:361-390). Only this and prepareFlush at flush time (:427-439) move a live DWPT's slice.
2. Each deleteDocuments(Term) appends a node and advances only the global slice (DocumentsWriterDeleteQueue.java:129-139, :198-215). Through Node.next (:411-412) the idle DWPT's head references every later node.
3. DocumentsWriter.applyDeleteOrUpdate (DocumentsWriter.java:153-165) calls flushControl.doOnDelete(), which passes no DWPT (DocumentsWriterFlushControl.java:497-501). In FlushByRamOrCountsPolicy.onChange every branch that marks a segment for flushing needs one, so only flushDeletes runs (FlushByRamOrCountsPolicy.java:40-57).
4. The push frees the global copy only. DocumentsWriterDeleteQueue.ramBytesUsed() is the global buffer (:557-560), reset at each push, and IndexWriter.ramBytesUsed() is that plus the DWPTs' own bytes (DocumentsWriterFlushControl.java:508-516, IndexWriter.java:795-799). None of these counts the chain, and no delete or update call flushes the idle segment; flush(), commit(), an NRT reopen, flushNextBuffer() or a later document in that segment move its slice. A later document does not free the span: finishDocuments copies it into the segment's own, counted pending deletes (DocumentsWriterPerThread.java:380-383, :718-722).

The same path serves deleteDocuments(Query...), updateNumericDocValue, updateBinaryDocValue, updateSortedNumericDocValue and updateDocValues(Term, Field...) (IndexWriter.java:1864-2075); the exception is a deleteDocuments(Query...) call with an exact MatchAllDocsQuery, which becomes deleteAll() (:1886-1890) and drops the buffered document too. Executed: Term deletes, numeric and sorted-numeric updates.

One buffered document on e1faeba08bde; the no-document column is the independent reproducer's control on 645d4fe14a20 (same fix-site code apart from #16867's global cap). These are measurements of this workload (OpenJDK 25.0.3, ByteBuffersDirectory, ids String.format("%016d", i)), not an allocation contract. updateNumericDocValue with one document: 51 to 253 MiB over 250,000 to 1.5 million updates, then OutOfMemoryError, ramBytesUsed() at most 13.9 MiB.

• Who is hit. Applications whose IndexWriter still holds at least one added document in RAM and then sends only deletes or doc-values updates for a long time without commit(), an NRT reopen or flush(): a bulk purge by id, a doc-values backfill, a tombstone replay. A buffered document is the normal state right after indexing.
• Default vs opt-in. Default IndexWriterConfig; nothing to opt into. One thread is enough.
• Scale. About 110-130 bytes of heap per 16-byte-id delete (127 in the probe), so roughly 8-10 million such calls per GB of heap. The growth is linear and ramBytesUsed() stays under the buffer, so an application that watches ramBytesUsed() (as the flushNextBuffer javadoc suggests, IndexWriter.java:3745-3757) cannot see it.
• How it ends. OutOfMemoryError, in application code or inside a writer call; in the latter case the writer treats it as tragic and closes, and uncommitted changes may be lost. No wrong results and no damage to the committed index.
• Why it is rare. Applications that refresh or commit periodically release the span each time, and nobody has reported it since 4.0 (2012). Workaround: flush(), commit() or a refresh every few hundred thousand deletes or updates.
• Not claimed. When the idle segment finally flushes it copies the whole pinned stream into its private buffer; at 2 GiB of delete terms that throws an "integer overflow" ArithmeticException from prepareFlush (a separate run with an 8 GB heap). Several idle segments after multi-threaded indexing pin the same nodes (no multiplication) but each copies the stream at flush (code reading only).

The smallest input: 100,000 deletes cross the 1 MB buffer at least twice (with the same buffer and ids, the diagram probe sees the first push at delete x44506), one StringField document is the least that leaves a segment buffered, and the weak reference to the first Term is the direct oracle for "still reachable after it was applied". Assertions on and off give the same failure. The same test is the regression test to add with a fix.

• On stock code the weak-reference check fails every time, since the GC never clears a strongly reachable referent; a fixed build passes only if System.gc() clears the reference (upstream removed a similar test in #12223). For a fix that flushes, assertEquals(0, w.numRamDocs()) after the loop is a GC-free alternative assertion.
• The comment line is longer than 100 columns, so ./gradlew tidy would reformat it.

# gradle, stock e1faeba08bde, seed 5EED1077, assertions on
TestIdleSegmentPinsAppliedDeletes > testBufferedDocumentPinsPushedDeletes FAILED
    java.lang.AssertionError: first delete term still reachable after 100,000 deletes expected null, but was:<id:x0>
        at org.apache.lucene.index.TestIdleSegmentPinsAppliedDeletes.testBufferedDocumentPinsPushedDeletes(TestIdleSegmentPinsAppliedDeletes.java:45)

# standalone JUnit runner, stock e1faeba08bde, assertions on and off: the same AssertionError
# control without addDocument, assertions off, 3 runs: pass
# with the recommended fix, gradle, seed 5EED1077, assertions on and off: TestIdleSegmentPinsAppliedDeletes, tests=1 failures=0

# supporting heap probe (not the repro): default config, -Xmx256m, assertions off, one buffered document; every sample line
mode=doc ops=250000 heapAfterGC=42MiB ramBytesUsed=7.1MiB numRamDocs=1
mode=doc ops=500000 heapAfterGC=83MiB ramBytesUsed=14.2MiB numRamDocs=1
mode=doc ops=750000 heapAfterGC=103MiB ramBytesUsed=6.6MiB numRamDocs=1
mode=doc ops=1000000 heapAfterGC=144MiB ramBytesUsed=13.8MiB numRamDocs=1
mode=doc ops=1250000 heapAfterGC=164MiB ramBytesUsed=6.1MiB numRamDocs=1
mode=doc ops=1500000 heapAfterGC=202MiB ramBytesUsed=13.2MiB numRamDocs=1
mode=doc ops=1750000 heapAfterGC=224MiB ramBytesUsed=5.6MiB numRamDocs=1
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

• The contract. LiveIndexWriterConfig.setRAMBufferSizeMB (LiveIndexWriterConfig.java:157-165 at e1faeba08bde): "Determines the amount of RAM that may be used for buffering added documents and deletions before they are flushed to the Directory" and "the writer will flush whenever buffered documents and deletions use this much RAM".
• No place in the internal accounting. DocumentsWriterFlushControl.getDeleteBytesUsed (:503-510): "heap bytes currently consumed by buffered deletes/updates that would be freed if we pushed all deletes". So it counts only what a push frees, the global buffer (a push does free it: in the heap probe ramBytesUsed() drops from 14.2 to 6.6 MiB between 500,000 and 750,000 deletes). The queue nodes an idle slice keeps are freed by no push, so they are counted nowhere. (Its next sentence excludes "already pushed delete/update packets", the frozen packets, not those nodes.)
• The hedges do not cover it. "just an indicator" is explained by concurrent flushes; the "only approximate" note names delete-by-Query only (LiveIndexWriterConfig.java:167-177). This is Term deletes, one thread, linear growth. The third note (:179-181: "only a subset of the buffered documents are flushed and therefore only parts of the RAM buffer is released") is about which buffered documents a flush writes; here no document is ever flushed, and the pinned span is never part of the counted buffer.
• Bounded without the document; released in 3.x. The no-document control stays at 10-22 MiB and the minimal test without its addDocument passes. In 3.1.0 and 3.6.2 the same workload flushes the buffered document together with the deletes and frees them.
• Precedents fixed as bugs. LUCENE-1717 (#2791, uncounted pending-delete RAM: "it's a trap, now. We shouldn't set traps."), LUCENE-3222 (delete-term undercount), LUCENE-3340 (#4413, buffered deletes not flushed by RAM, called a memory leak, its tests "fail on trunk and pass on 3.x"), LUCENE-9478 (#10517, "DWPTDeleteQueue leaks memory", a delete-queue retention leak by another mechanism) and LUCENE-9298 (#10338, RAM accounting in BufferedUpdates when deleted doc IDs and terms are cleared).
• Design discussion. On #3647 (LUCENE-2573, 2011) McCandless wrote that the flush policy "needs access to the 'global' deletes count (and also the to [sic] per-DWPT deletes count). If a given DWPT has 0 buffered docs, then indeed the buffered deletes in its pool doesn't matter". Here it has one.
• No test asserts the retention. TestIndexWriterDelete.testIndexingThenDeleting, testFlushPushedDeletesByRAM and testApplyDeletesOnFlush never leave an idle buffered document; the numRamDocs() > 0 asserts of TestPerSegmentDeletes are in a commented-out block (:102-158).

The strongest case for "intended" (all the reviews from another model family rated it moderate; the devil's-advocate verifier called the javadoc sentence the main risk, with medium confidence on how upstream will receive it): the IndexWriter class javadoc says "In contrast to the other flush options setRAMBufferSizeMB and setMaxBufferedDocs, deleted terms won't trigger a segment flush" (IndexWriter.java:129-131); the 4.0 CHANGES entry says "IndexWriter only applies and flushes deletes if the maxBufferedDelTerms limit is reached during indexing. No segment flushes will be triggered due to this setting." (lucene/CHANGES.txt:15781-15783); the dev@ thread "FlushPolicy and maxBufDelTerm" (2013-08-01) accepted that delete pressure does not write a segment; the kept deletes really are needed until the segment flushes; and the RAM limit is "just an indicator".

Why it fails: the javadoc sentence was added in f47c3e82985 (LUCENE-3023, 2011-04-28) right after a sentence about setMaxBufferedDeleteTerms, contrasting that count setting with the RAM buffer; 58105a203a1 (LUCENE-7868, 2017) removed the count sentence and left this one. The CHANGES entry and the 2013 thread are about the same count setting ("due to this setting"), and in that thread McCandless wrote "if RAM usage of the buffered del Term/Query s is too high, then a segment is flushed along with the deletes being applied". Nothing documents unbounded, uncounted retained heap; "needed until the flush" is why the slice exists, not a reason never to flush or count it. The fix does not need a segment flush per delete (see Suggested fix).

Option A was written for this doc and validated through gradle in a scratch worktree of e1faeba08bde (OpenJDK 25.0.3). Option C is the hunt's earlier sketch; B and D were not implemented.

DocumentsWriter.applyDeleteOrUpdate (DocumentsWriter.java:153-165) keeps its synchronized body as applyDeleteOrUpdateLocked; when that returns a negative sequence number (this call pushed the global buffer, :161-163), it flushes, outside the lock, every DWPT returned by a new DocumentsWriterFlushControl.checkoutIdleWriterWithDocs(). That method takes only DWPTs that have documents, are not flush-pending and can be tryLocked, marks one pending and checks it out, like checkoutLargestNonPendingWriter. The flushed segment is published by the caller's IndexWriter.maybeProcessEvents, which already runs for a negative sequence number (IndexWriter.java:6345-6351). +45/-1 in DocumentsWriter.java and DocumentsWriterFlushControl.java.

private long applyDeleteOrUpdate(ToLongFunction<DocumentsWriterDeleteQueue> function)
    throws IOException {
  final long seqNo = applyDeleteOrUpdateLocked(function); // the old synchronized body
  if (seqNo < 0) { // this call pushed the global delete buffer
    DocumentsWriterPerThread idle;
    while ((idle = flushControl.checkoutIdleWriterWithDocs()) != null) {
      doFlush(idle); // prepareFlush applies and drops the idle slice
    }
  }
  return seqNo;
}

Trade-offs

• Bound. For every idle DWPT the push can lock, the kept span ends at that push, so it stays within about one RAM buffer of deletes. Skipped DWPTs keep theirs: one whose tryLock() fails (an indexing thread holds it; its slice moves at its next document) or one already flush-pending. Nothing retries them, so a skipped DWPT that then stays idle keeps its span until a later push finds it free. The span is also still uncounted before the first push: in the heap probe the heap reached 83 MiB at 500,000 deletes (16 MB buffer, almost 6 times the counted 14.2 MiB) before the first push released it.
• Cost. A delete or update call that pushes may now write segments, so it does flush I/O and can throw what a flush throws, as addDocument already can. The flush runs after the delete is queued and pushed, so an exception from it reaches the caller of a delete that has already taken effect; that needs a decision upstream. Each push flushes each idle DWPT it can lock once, and doFlush then also flushes any other DWPT already pending (DocumentsWriter.java:502-580), as the add path does. An idle DWPT's segment holds whatever was buffered when indexing paused; segment sizes and counts were not measured, and no throughput benchmark was run. It happens only when the global delete buffer alone reaches the RAM buffer; with documents being added, the add path's activeRam + deletesRam check usually flushes first (FlushByRamOrCountsPolicy.java:54-55, by reading). With RAM flushing disabled (DISABLE_AUTO_FLUSH), pushes happen only at #16867's 1945 MB cap, so the fix would act only there (by reading).
• Concurrency. A DWPT held by an indexing thread is skipped. A push never happens during a full flush (DocumentsWriter.java:175-176). If a full flush begins between the push and the checkout, checkout marks nothing: it returns null for a DWPT that is not flush-pending, and for one that is it blocks it and returns the next pending flush (DocumentsWriterFlushControl.java:261-280), the same handling checkoutLargestNonPendingWriter (:789-810, used by flushNextBuffer()) relies on; the pending flag is re-read under the flush-control lock. No concurrency or full-flush test of this path was run.
• Docs. The IndexWriter javadoc sentence "deleted terms won't trigger a segment flush" would need updating.

What the checks validated

• The minimal test fails on stock e1faeba08bde (gradle, seed 5EED1077) and passes with the fix, assertions on and off (gradle on a copy of e1faeba08bde with the diff applied; the JUnit XML names TestIdleSegmentPinsAppliedDeletes; the fact check's standalone JUnit runs agree).
• The independent root-cause reviewer's 3-test class (Term deletes, the no-document control and updateSortedNumericDocValue, which option C misses) passes with the fix.
• 17 closest suites (TestIndexWriterDelete, TestFlushByRamOrCountsPolicy, TestDocumentsWriterDeleteQueue, TestDocumentsWriterStallControl, TestNumericDocValuesUpdates, TestBinaryDocValuesUpdates, TestMixedDocValuesUpdates, TestIndexWriter, TestIndexWriterExceptions, TestIndexWriterWithThreads, TestIndexWriterReader, TestStressDeletes, TestIndexWriterThreadsToSegments, TestPerSegmentDeletes, TestIndexWriterMaxDocs, TestForceMergeForever, TestSoftDeletesRetentionMergePolicy): 356 tests, 16 skipped, 0 failures, seeds 5EED1077 and C0FFEE77.
• checkGoogleJavaFormat reports nothing in the two changed files; ecjLintMain passes.
• Heap probe with the fix: Term deletes and updateNumericDocValue both complete 2.5 million calls in a 256 MB heap, 5-21 MiB after the first push, numRamDocs() 0.

Track the bytes appended to the queue since each DWPT's slice head (for example a running total of the global buffer's bytes at each push, recorded when a slice moves), add it to getDeleteBytesUsed(), and let FlushByRamOrCountsPolicy flush or apply the idle DWPT once it passes the buffer.

Trade-offs

More precise than A: it also bounds the uncounted span before the first push and makes ramBytesUsed() honest. More code on hot paths, and it needs a per-node byte estimate.

while (docWriter.flushOneDWPT()) {} after the push in five IndexWriter delete and update methods.

Trade-offs

Validates the root cause (the minimal test passes with assertions on and off), but misses updateSortedNumericDocValue, and flushOneDWPT takes the largest DWPT with a blocking lock, including ones indexing threads are using; the adjudication's summary of the reviews calls it too blunt (tiny segments).

Say in setRAMBufferSizeMB and the IndexWriter class javadoc that while a document is buffered, every later delete and update is kept until the next flush, commit or refresh, and recommend flush() during long delete-only or update-only runs.

Trade-offs

Cheapest; leaves the trap.

Method: non-formal (a measurement first, then code reading; the later bounded enumeration is semi-formal).

• The hunt (2026-11-02) ran a heap-versus-RAM-buffer probe on the stock main jar: default config, one thread, -Xmx256m, one buffered document, then N deleteDocuments(Term) or updateNumericDocValue calls, sampling heap after GC and ramBytesUsed() every N/8 calls, against a control with nothing buffered. The idle run grew to OutOfMemoryError; the control stayed bounded.
• A hunt unit read the DeleteSlice code for the mechanism and reduced it to one public-API JUnit test.
• It then checked the prediction with a bounded model-based enumeration (every sequence of up to 3 of 8 writer operations before a delete or doc-values burst, 1,170 runs, assertions on and off): 380 runs kept the first term, all exactly those with numRamDocs() > 0 when the burst started, 0 mismatches. With its fix sketch as a vacuity control, 0 runs kept it.
• Commit-level bisection and version ports traced it to the 4.0 per-thread writer landing.