updateDocuments with an empty block leaks about 33 KB of IndexWriter RAM accounting per update-and-refresh round, until the writer flushes after every new document

IndexWriter.updateDocuments(term, block) and softUpdateDocuments(term, block, softField) accept an empty block: delete or soft-delete by key, add nothing. The delete applies, but the call still takes a per-thread writer (DWPT) and stores the key in that writer's private buffer, so the flush control adds 33,040 bytes (32,960 on the soft path) to its running RAM total. The next full flush (an NRT reopen, commit() or flush()) finds the writer with 0 documents and "drop[s] it on the floor" in DocumentsWriterFlushControl.markForFullFlush without subtracting those bytes; every other exit (flush, abort) subtracts them.

So ramBytesUsed() grows by about 33 KB per such round (an empty-block update, then a full flush) and never comes down for the life of the writer, even past the real heap. Once the phantom total reaches the RAM buffer (16 MB by default, about 500 rounds), the flush policy fires after every added document, so in a single-threaded writer each document becomes its own segment, with the merge and I/O work that follows. No hit is wrong and nothing on disk is malformed; closing the writer clears it. With assertions on, one round makes close() fail and, at the buffer, the writer closes itself. It dates from 4.5.0 (2013); since 9.9.0 (and 7.7.0 on the soft path) each round leaks about 33 KB instead of 165-330 bytes. A four-line fix in the drop branch releases the bytes.

1. The call. IndexWriter.updateDocuments(Term, Iterable) (IndexWriter.java:1593-1610) wraps the term in a delete node and calls DocumentsWriter.updateDocuments; softUpdateDocuments (:1644-1675) does the same with a doc-values update node. It rejects a null term and missing soft-delete fields (:1667-1672), not an empty block.
2. A writer is taken anyway. DocumentsWriter.updateDocuments (DocumentsWriter.java:412-454) always calls flushControl.obtainAndLock() (:418).
3. The key lands in the writer's private buffer. In DocumentsWriterPerThread.updateDocuments (DocumentsWriterPerThread.java:245-301) the loop over the block runs zero times, so the writer keeps 0 documents. finishDocuments (:361-390) adds the delete to the global queue and applies it to the writer's own pendingUpdates with docIdUpTo = 0 (:374-377): it can delete nothing here, but the term is stored. The first term allocates the 32 KB first block of BufferedUpdates.DeletedTerms' ByteBlockPool (BufferedUpdates.java:183-184, ByteBlockPool.BYTE_BLOCK_SIZE = 1 << 15) since #12573 (9.9.0); on the soft path FieldUpdatesBuffer accounts a 32 KB block since LUCENE-8590 (7.7.0). Measured: 33,040 bytes (hard) and 32,960 (soft) per round.
4. The bytes are charged. DocumentsWriterFlushControl.doAfterDocument (DocumentsWriterFlushControl.java:207-250) skips deltas below 16 KB (:211-215); 33,040 bytes is above that, so it commits them to the writer (:225) and adds them to activeBytes (:236). The writer goes back to the pool with 0 documents.
5. The full flush forgets them. A refresh, commit or flush calls markForFullFlush (:557). A writer with documents is marked pending, which moves its bytes to flushBytes (:365-376), and doAfterFlush subtracts them once the segment is written (:290-295). A writer with 0 documents takes the branch at :596-602, "In this case we simply remove it from the pool and drop it on the floor": perThreadPool.checkout(next) and nothing else. doOnAbort (:378-389) shows the intended bookkeeping: it subtracts getLastCommittedBytesUsed().
6. Nobody can take them back. The dropped writer is out of the pool, so no later flush, abort or deleteAll() sees it (executed in the fact check: 33,040 bytes remain after a further commit() and after deleteAll()). No operation on the still-open writer repairs the total; rollback() closes the writer and, under -ea, hits the same netBytes assertion (by reading). ramBytesUsed() (IndexWriter.java:796-799 -> DocumentsWriterFlushControl.java:501-504) keeps the bytes until the writer is closed.
7. The flush storm. FlushByRamOrCountsPolicy.onChange (FlushByRamOrCountsPolicy.java:34-53) flushes the largest writer when activeRam + deletesRam >= limit (:49-50). Once the phantom bytes alone reach the limit, that holds after every added document, so every document triggers a flush of the largest non-pending writer that holds documents (DocumentsWriterFlushControl.java:749-772). In a single-threaded workload that is the writer holding just the new document: one segment per document. With several indexing threads it can be another, larger writer (by reading).
8. With -ea. The next empty-block update after that point reaches assert perThread.getNumDocsInRAM() > 0 (FlushPolicy.java:74); IndexWriter treats the AssertionError as tragic and closes itself. The message users see, netBytes has unexpected value 0 != N, is the rollback's own check (DocumentsWriter.java:232-233), which also fails close() after a single round.

Precondition, exactly: the per-thread writer that took the empty update still has 0 documents at the next full flush. In a single-threaded workload a document added before the refresh lands in the same writer and nothing leaks; with concurrent indexing it can leak while other writers hold documents. Several empty updates between two full flushes reuse one writer, so the leak counts full flushes, not calls. Other entry points (executed in the fact check): a zero-row column batch, updateDocuments(Term, ColumnBatch), leaks the same 33,040 bytes through DocumentsWriter.updateBatch. updateDocuments(Query, List.of()) takes the same path, but a delete-by-Query is accounted at about 76 bytes, below the 16 KB granularity skip, so with the default config it leaves 0 bytes (76 per round with setMaxBufferedDocs).

• Who is hit. Code that calls updateDocuments(term, block) or softUpdateDocuments(term, block, ...) with a block that can be empty. The default IndexWriterConfig, codec, flush policy and merge policy suffice. The block APIs are @lucene.experimental; softUpdateDocuments is documented as "Expert".
• Scale. About 33 KB per round (an empty-block update, then a refresh or commit) since 9.9.0 (updateDocuments) and 7.7.0 (softUpdateDocuments): 16,777,216 / 33,040 = 508 rounds fill the default buffer (the -ea failure came at rounds 506-509). Older releases leak 165 or 330 bytes per round: on 4.5.0, 110,000 rounds left 18,150,000 bytes and 200 documents then became 200 segments.
• Assertions off (the JVM default). ramBytesUsed() over-reports without bound and past the real heap (with a 2 GB buffer: 268,450,000 bytes on a 268,435,456-byte max heap after 8,125 rounds); the bytes are phantom, no heap is held. Past the buffer every added document triggers a flush; in these single-threaded runs, after 600 rounds, 200 small documents caused 200 flushes with the stock merge policy, and with NoMergePolicy 100 documents became 100 segments. Merge and I/O load follow; throughput was not measured. Code that drives its own flushing from ramBytesUsed(), as the flushNextBuffer() javadoc invites (IndexWriter.java:3752-3753), is misled the same way; getFlushingBytes() is not inflated (by reading).
• Not affected. Deletes and soft deletes still apply, no hit is wrong, segments are written by the normal flush path (by reading; CheckIndex not run), and no hang was found (by reading the stall condition). Closing the writer and opening a new one clears it. By reading (not executed), back-pressure does change: stall control holds indexing threads only while activeBytes is below twice the RAM buffer (DocumentsWriterFlushControl.java:318), so once the phantom bytes alone reach that (about 1,016 rounds at 16 MB), slow flushes can no longer stall indexing for the rest of the writer's life.
• Assertions on (tests, development). One empty-block update plus a refresh or commit makes close() throw; at the buffer the writer closes itself tragically and loses what was buffered. An application's own tests under Lucene's test framework would see the first symptom.
• Not hit. In their default indexing paths (read from their source, not run; not a claim about every extension or branch), Elasticsearch and OpenSearch call softUpdateDocuments only for more than one document, and Solr always passes at least the root document. In Lucene no non-test code calls these methods except IndexWriter, and no upstream test passes a literal empty block.

Fully default IndexWriterConfig, ByteBuffersDirectory; the test itself uses no randomness (the test framework still randomizes the default codec and similarity; the result is the same at seeds DEADBEEF and 5EED1234). It asserts only the accounting: after a refresh nothing is buffered, so ramBytesUsed() must be 0, as it is after deleteDocuments(term) and with the fix. The growth over rounds and the flush storm are consequences of the same leak and are not re-shown; the delete still applying, the soft path and the segment counts rest on the other checkers' runs, not on this test. It sits in org.apache.lucene.index only because that is where an upstream test would go. As a regression test it is the same test (for example in TestFlushByRamOrCountsPolicy or TestIndexWriter); it fails on stock and passes with the fix. All runs were on Linux aarch64 with OpenJDK 25.0.3; the missing subtraction is platform-independent, the exact byte count was measured only there.

# gradle, stock 645d4fe14a20, seed DEADBEEF, assertions on
TestEmptyBlockUpdateRamAccounting > testEmptyBlockUpdateReleasesRamOnRefresh FAILED
    java.lang.AssertionError: expected:<0> but was:<33040>
    Suppressed: java.lang.AssertionError: netBytes has unexpected value 0 != 33040   (from close())

# assertions off (-Ptests.asserts=false): the same message, close() succeeds
# 10.6.0 release jars, assertions on and off: the same message (suppressed close() error with -ea only)

# with the fix: 1 test, 0 failures, assertions on and off

• Accepted input. The javadocs of updateDocuments (IndexWriter.java:1593-1604) and softUpdateDocuments (:1644-1663) do not require a non-empty block; softUpdateDocuments validates its other arguments (:1667-1672) but not this one. addDocuments(List.of()) takes the same path with no delete and leaves 0 bytes (executed controls).
• The RAM buffer's contract. LiveIndexWriterConfig.setRAMBufferSizeMB: "When this is set, the writer will flush whenever buffered documents and deletions use this much RAM" (LiveIndexWriterConfig.java:162-163).
• Lucene's invariants. DocumentsWriter.abort asserts flushControl.netBytes() == 0 (DocumentsWriter.java:232-233); TestFlushByRamOrCountsPolicy asserts flushControl.activeBytes() == 0 after writer.close() (TestFlushByRamOrCountsPolicy.java:102-103); setFlushPending, doAfterFlush and doOnAbort all subtract a writer's committed bytes. The drop branch is the only exit that does not.
• The documented slack is bounded and in the other direction. Deletes by Query are under-counted (LiveIndexWriterConfig.java:174-177), and the 16 KB granularity skip is bounded by design: "we'd never be missing more than 64*16kB = 1MB" (DocumentsWriterFlushControl.java:201-202). This is an unbounded, permanent over-count.
• Precedent. #5464 (LUCENE-4398, 2012): a different accounting leak with the same symptom, "every single document being flushed into its own segment", was treated as a bug and fixed with a test.

The strongest case for "intended": the block APIs are experimental and softUpdateDocuments is Expert, a pure delete should use deleteDocuments or updateDocValues, the RAM buffer is "just an indicator when to flush" (LiveIndexWriterConfig.java:167-172), the drop branch's comment says a 0-document writer only comes from a concurrent flush, and internal code treats 0-document writers as non-flushable (the setFlushPending javadoc: "must have indexed at least on Document" [sic], DocumentsWriterFlushControl.java:360-364; FlushPolicy.java:74 and DocumentsWriterPerThread.prepareFlush, :428, assert it).

Why it fails: nothing forbids or rejects the empty block, and the experimental tag is about API stability, not about the writer's accounting drifting; the "indicator" sentence says the writer may use more memory than the buffer, not that the total may grow forever, and here it passes the real heap; the comment shows the empty block was not foreseen, it is not a user-facing precondition. The internal assertions are assumptions, not a documented restriction on callers; they are a fair argument for rejecting empty blocks or skipping the writer (the other two options under Suggested fix) rather than for the leak. In 2020 mikemccand asked on apache/lucene-solr#1397, the PR that wrote today's branch, "Hmm why would we ever have nextRam > 0 but next.getNumDocsInRAM() == [0]?" (the [0] is ours; the comment ends at ==), and nobody replied; an empty-block update is one concrete way to reach that state. Every checker rated the case weak: real, minor.

The hunt's fix, unchanged (an independent checker wrote the same change), run through gradle in a scratch worktree of 645d4fe14a20 (OpenJDK 25.0.3, seed DEADBEEF). It applies to upstream main after #16867 with a 12-line offset. The other two options are alternatives maintainers may prefer; neither was implemented.

In the 0-document branch of DocumentsWriterFlushControl.markForFullFlush (:596-602), subtract next.getLastCommittedBytesUsed() from activeBytes under the flush-control lock, then check assertMemory(). A 0-document writer is never flush-pending (setFlushPending acts only when getNumDocsInRAM() > 0, :367), so its bytes are in activeBytes, and the writer is locked here. The delete itself is unaffected: it is in the global delete queue, and the private copy could only apply to this writer's documents, of which there are none. Because it fixes the drop site, it covers (by reading) any 0-document writer that holds committed bytes, including the zero-row column-batch route (which leaks the same 33,040 bytes, executed in the fact check) and, with setMaxBufferedDocs, the Query routes.

         } else {
           // it's possible that we get a DWPT with 0 docs if we flush concurrently to
-          // threads getting DWPTs from the pool. In this case we simply remove it from
-          // the pool and drop it on the floor.
+          // threads getting DWPTs from the pool, or after an update with an empty block of
+          // documents. In this case we simply remove it from the pool and drop it on the floor,
+          // releasing whatever RAM it already accounted for.
           boolean checkout = perThreadPool.checkout(next);
           assert checkout;
+          synchronized (this) {
+            activeBytes -= next.getLastCommittedBytesUsed();
+            assert assertMemory();
+          }
         }

Trade-offs

No correctness cost known. One more short lock section per dropped writer during a full flush, which is rare; not benchmarked. The full ./gradlew check was not run.

What the checks validated

• The minimal repro fails on stock and passes with the fix, assertions on and off.
• The finder's 4-test class (both entry points over 100 commit rounds at a 1 MB buffer, 200 documents in at most 3 segments, close() after one round, the addDocuments(List.of()) control): 4 of 4 pass, assertions on and off.
• The 11 closest suites (TestIndexWriter, TestIndexWriterDelete, TestFlushByRamOrCountsPolicy, TestDocumentsWriterDeleteQueue, TestDocumentsWriterStallControl, TestDocumentsWriterPerThreadPool, TestIndexWriterExceptions, TestIndexWriterThreadsToSegments, TestSoftDeletesRetentionMergePolicy, TestPendingSoftDeletes, TestSoftDeletesDirectoryReaderWrapper): 233 tests, 0 failures, 11 skipped.
• checkGoogleJavaFormat, ecjLintMain, ecjLintTest, forbiddenApisMain and forbiddenApisTest of :lucene:core pass (the separate editorconfig linter, eclint, is not configured there and did not run).
• Earlier: a fix of the same shape on 4.5.0 removes the leak there (0 bytes after 110,000 rounds, 200 documents in 1 segment).

Put the delete or doc-values update only into the global delete queue, as deleteDocuments and updateDocValues do.

Trade-offs

Avoids the 32 KB allocation altogether. But the block is an Iterable: detecting "empty" means taking its iterator before obtaining a writer and handing that same iterator on, plus a second sequence-number path, and the column-batch route needs the same. It leaves the drop branch unguarded for any other source of a 0-document writer with bytes. Not implemented or tested.

One check in each API, which also documents the restriction.

Trade-offs

Changes accepted behaviour: addDocuments(List.of()) is accepted today, and callers that pass an empty block to mean "delete only" would start failing. Leaves the drop branch unguarded. Not implemented or tested.

Method: non-formal (probes and executed tests; no model checker, solver or proof produced the signal), a hunt unit on soft deletes.

• The finder was extending a model-based test of soft deletes to softUpdateDocuments edge cases. A probe (softUpdateDocuments(term, List.of(), soft), then close()) failed under -ea in DocumentsWriter.abort.
• Two more probes then measured, with assertions off, 660,800,000 bytes after 20,000 rounds on a 64 MB heap, and, with a 1 MB RAM buffer and NoMergePolicy, 201 segments for 200 small documents after 100 rounds (2 in the controls).
• A JUnit test on the stock jars, with assertions on and off, became the candidate; an independent reproducer, a devil's advocate, read-only reviews by another model family and an adjudication then checked it, and the minimal test was cut from the adjudication's test.