During a commit or NRT refresh, deleteDocuments(Term...) and doc-values updates never wait on the delete-buffer stall added by #16867, so the shared delete buffer still grows past its hard limit and can overflow at 2 GiB (main and branch_11x only)

Finding LFV-1076Area: index / IndexWriter delete buffer and stall control (DocumentsWriter.applyDeleteOrUpdate vs the #16867 delete-buffer hard limit)Priority: lowImpact: index unusableNovelty: known upstream

AI-assisted finding, confirmed by automated checks, not yet triaged by a human Lucene committer. This site files nothing upstream.

What it shows

On Lucene main with #16867, one thread runs a commit or NRT refresh while another calls deleteDocuments(Term...) or a doc-values update (updateNumericDocValue, updateBinaryDocValue, updateDocValues). You expect these calls to be held at #16867's new 'hard limit' on the shared delete buffer, as document adds and updates (softUpdateDocument included) are, so that the buffer can no longer overflow. Instead only document adds and updates wait on the new stall: the delete and doc-values-update calls keep filling the buffer (to 8x a cap lowered to 256 KB in the repro), and at the real 1945 MB cap they still overflow it at 2 GiB with ArithmeticException 'integer overflow', after which the writer fails every flush and commit until rollback().

Expected vs actual

• Expected: You expect these calls to be held at #16867's new 'hard limit' on the shared delete buffer, as document adds and updates (softUpdateDocument included) are, so that the buffer can no longer overflow.
• Actual: Only document adds and updates wait on the new stall: the delete and doc-values-update calls keep filling the buffer (to 8x a cap lowered to 256 KB in the repro), and at the real 1945 MB cap they still overflow it at 2 GiB with ArithmeticException 'integer overflow', after which the writer fails every flush and commit until rollback().

Files

• src/org/apache/lucene/index/TestDocValuesUpdateBufferHardLimitDuringFullFlush.java (JUnit test)

Run it

• Tested on Lucene main e1faeba08bde (2026-11-02): still reproduces.
• Tested on Lucene main f815e679bf6 (2026-11-01): reproduced.
• How to read the outcome: a failing test is the finding reproducing (the failure message states expected against actual).
• Seeds: a rerun with the same seed replays the same random choices. Assertions off (-da -dsa -Dtests.asserts=false) is the production-like setting.

# Lucene checkout at the tested commit, built with: ./gradlew --no-daemon assemble   (JDK 25)
LUCENE=/path/to/lucene; V=12.0.0-SNAPSHOT
CP=$LUCENE/lucene/core/build/libs/lucene-core-$V.jar
CP=$CP:$LUCENE/lucene/codecs/build/libs/lucene-codecs-$V.jar
CP=$CP:$LUCENE/lucene/test-framework/build/libs/lucene-test-framework-$V.jar
CP=$CP:randomizedtesting-runner-2.9.2.jar:junit-4.13.2.jar:hamcrest-3.0.jar   # test-framework dependencies (Maven Central)
CP=$CP:junit-platform-console-standalone-6.1.3.jar   # JUnit Platform console launcher (Maven Central)

javac -d classes -cp "$CP" $(find src -name '*.java')

# run 1: assertions off (production-like), seed DEADBEEF
java -da -dsa -Dtests.asserts=false --add-modules jdk.incubator.vector --enable-native-access=ALL-UNNAMED -Dtests.seed=DEADBEEF \
     -cp "classes:$CP" org.junit.platform.console.ConsoleLauncher \
     execute --select-class org.apache.lucene.index.TestDocValuesUpdateBufferHardLimitDuringFullFlush \
       --select-class org.apache.lucene.index.TestDocValuesUpdateBufferHardLimitDuringFullFlush

Gradle alternative: copy the test into the matching module's src/test/ tree of the checkout and run ./gradlew -p lucene/<module> test --tests <class> -Ptests.seed=<seed> (add -Ptests.asserts=false for a run without assertions).

Feedback

Open an issue in lucene-ai-bug-hunt. Please do not open issues or pull requests in apache/lucene about this finding unless a Lucene committer agrees it should be filed.

Snapshot of 2026-11-04. AI-assisted; not yet triaged by a human Lucene committer. Feedback: open an issue in lucene-ai-bug-hunt.