A rejected over-long delete term stays in IndexWriter's delete queue, so later adds, deletes and the next commit fail

IndexWriter.deleteDocuments(Term...), updateDocument(Term, ...) and updateDocuments(Term, ...) link the delete into the writer's shared delete queue before they apply it. Since #12573 (9.9.0) applying it stores the term in a BytesRefHash, which rejects keys over 32,766 bytes, the same limit IndexWriter.MAX_TERM_LENGTH puts on indexed terms. A delete by a 32,767-byte term therefore throws MaxBytesLengthExceededException after it is queued, and because the queue's slices are reset only after a fully successful apply, the bad entry is applied again, and fails again, by the next addDocument, delete, commit, NRT reopen or close(). The documents buffered before it are lost: rollback() discards everything since the last commit, and a commit retried after the failed one succeeds without them; committed data stays intact. Up to 9.8.0 the same call was a no-op, and on main the same term as deleteDocuments(new TermQuery(term)) still is. Default settings, one call, one thread; every release from 9.9.0 to 10.6.0 and main are affected.

1. Queued before it is checked. deleteDocuments(Term...) (IndexWriter.java:1864-1872) reaches DocumentsWriterDeleteQueue.addDelete, which does long seqNo = add(new TermArrayNode(terms)); tryApplyGlobalSlice(); (DocumentsWriterDeleteQueue.java:129-133): the entry is linked at the tail of the shared queue and gets a sequence number first.
2. The apply throws. tryApplyGlobalSlice (:198-215) applies the global slice; DeleteSlice.apply (:348-367) calls each entry's apply; TermArrayNode.apply (:480-485) calls BufferedUpdates.addTerm, which ends in deleteTerms.put(term, docIDUpto) (BufferedUpdates.java:131) -> DeletedTerms.put (:210-221) -> BytesRefHash.add (:325-327) -> BytesRefBlockPool.addBytesRef, which throws MaxBytesLengthExceededException: bytes can be at most 32766 in length; got 32767 (BytesRefBlockPool.java:83-90).
3. Never dropped. DeleteSlice.apply calls reset() only after its loop finished (DocumentsWriterDeleteQueue.java:366). The exception skips it, so the global slice, and every per-thread slice that later reaches the entry, starts before it again next time and throws at the same entry.
4. Later calls. The next addDocument applies its thread's slice in DocumentsWriterPerThread.finishDocuments (:380-383) and throws; the next delete or updateDocument applies the global slice again (DocumentsWriterDeleteQueue.java:131, :168 -> tryApplyGlobalSlice, :209) and throws.
5. Commit. DocumentsWriter.flushAllThreads (:667-715) swaps in a new queue and flushes. DocumentsWriterPerThread.prepareFlush (:427-438) first extends the thread's slice to the queue's tail (DocumentsWriterDeleteQueue.java:229-232; the global slice is not applied again there, since the failed apply already set its tail to the bad entry, :264, :296-300) and then applies that slice (DocumentsWriterPerThread.java:434), which reaches the bad entry and throws. The finally then calls close() on the old queue (DocumentsWriter.java:704-708), which sees unapplied changes and throws IllegalStateException: Can't close queue unless all changes are applied (DocumentsWriterDeleteQueue.java:317-321), replacing the original exception; with assertions off the writer stays open. With assertions on, the commit ends in an AssertionError ("Can not replace a full flush queue if the queue is not closed", DocumentsWriter.java:647-649), which prepareCommitInternal treats as tragic (IndexWriter.java:3879-3881), and the writer closes itself.

Document a committed, b added, then the 32,767-byte delete: the delete and addDocument(c) throw MaxBytesLengthExceededException, an NRT reopen throws the IllegalStateException, and after that the old queue has been dropped, so commit() and close() return normally. The committed index holds 1 document, a; b is lost (c was rejected), and hasUncommittedChanges() stays true. With nothing committed before, 0 of 3. The same at the 10.6.0 release commit. On 9.8.0 the same sequence commits all 3. close() right after the bad delete throws the same IllegalStateException and closes the writer without committing b (1 of 2).

All runs were single-threaded. The global apply runs only when its lock is free (DocumentsWriterDeleteQueue.java:199), so with concurrent writers the bad call itself may return normally and a later call, or the next commit, hit the entry instead (by code reading).

Up to 9.8.0 BufferedUpdates kept delete terms in a HashMap<Term, Integer> with no key-length limit, so applying a term entry could not throw for this reason. The queue behaviour (append before apply; reset only after a full apply) is from 2011 (LUCENE-2956) and 2016; #12573 gave a legal public call a way to make that step throw.

• Trigger. One call of deleteDocuments(Term...), updateDocument(Term, ...) or updateDocuments(Term, ...) with a term of at least 32,767 bytes. Default codec and IndexWriterConfig, any directory (reproduced on ByteBuffersDirectory, FSDirectory and the test framework's directory); one thread. In the monitor module, Monitor.deleteById passes the id straight to deleteDocuments(new Term(query_id, id)): after deleteById of a 32,767-character id the next register() and Monitor.close() throw on main, while 9.8.0 registers both queries.
• Effect. Later adds and deletes throw and the next commit, NRT reopen or close() fails. The documents buffered before the bad delete are lost: rollback() discards all uncommitted work, and a commit retried after the failed one succeeds without them (after that failed commit the writer works again). The failure is loud (every affected call throws). With assertions off the writer is not marked tragic; with assertions on the failed commit closes it. Committed data is not damaged: the committed index opens and a new writer opens on it in every probe. A secondary consequence: the dropped per-thread writer keeps its 4 stored-fields files open, still after rollback() and Directory.close().
• Not affected. The doc-values update APIs (updateNumericDocValue, updateBinaryDocValue, updateDocValues; executed), whose buffer has no such key limit; softUpdateDocument(s), which uses that same buffer (executed: softUpdateDocument with the 32,767-byte term and then commit() both succeed); deleteDocuments(Query...) with a TermQuery of the same term (executed: a no-op).
• Versions. Every release from 9.9.0 to 10.6.0, and main.
• Likelihood: rare. Real ids are never that long, so it takes an unvalidated oversized key; but a single such call breaks the writer's next adds, deletes and commit and loses its buffered documents, for every client of that writer, not just the bad request.

The test sits in Lucene's core index test package. The input is fixed. Through gradle the seed also decides whether assertions are on unless -Ptests.asserts is given (on for about 75% of seeds; DEADBEEF turns them on); otherwise it picks only the test framework's random directory wrapper, locale and codec settings, which the test does not depend on. The bug also reproduces without the test framework, in plain programs on ByteBuffersDirectory and FSDirectory.

Through the test framework's directory wrapper the failure also carries a suppressed note that 4 stored-field files of the dropped per-thread writer are still open. The wrapper only detects this: a plain FSDirectory program shows the same 4 descriptors still open after rollback() and Directory.close(). A secondary consequence, not the headline.

Regression test: this same test; it passes with either fix below. Before it goes into a PR, ./gradlew tidy wraps its one comment line (101 characters), which checkGoogleJavaFormat flags.

cp $REPRO/TestOverlongDeleteTermKeepsWriterUsable.java $LUCENE/lucene/core/src/test/org/apache/lucene/index/
cd $LUCENE && ./gradlew :lucene:core:test --tests TestOverlongDeleteTermKeepsWriterUsable -Ptests.seed=DEADBEEF
# assertions off: add -Ptests.asserts=false

# main 645d4fe14a20, gradle, -Ptests.seed=DEADBEEF -Ptests.asserts=false
TestOverlongDeleteTermKeepsWriterUsable > testCommitAfterOverlongDeleteTerm FAILED
    java.lang.IllegalStateException: Can't close queue unless all changes are applied
        at org.apache.lucene.index.DocumentsWriterDeleteQueue.close(DocumentsWriterDeleteQueue.java:321)
        at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:707)
        at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3809)
        at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:4253)
        at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4215)
        at org.apache.lucene.index.TestOverlongDeleteTermKeepsWriterUsable.testCommitAfterOverlongDeleteTerm(TestOverlongDeleteTermKeepsWriterUsable.java:39)

# main 645d4fe14a20, gradle, -Ptests.seed=DEADBEEF (assertions on)
TestOverlongDeleteTermKeepsWriterUsable > testCommitAfterOverlongDeleteTerm FAILED
    java.lang.AssertionError: pendingNumDocs 1 != 0 totalMaxDoc
        at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2596)
        at org.apache.lucene.index.IndexWriter.maybeCloseOnTragicEvent(IndexWriter.java:6104)
        at org.apache.lucene.index.IndexWriter.tragicEvent(IndexWriter.java:6094)
        at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3880)
        ...

# plain step probe, main, assertions off (9.8.0: every step ok, 3 documents committed)
  deleteDocuments(huge): threw MaxBytesLengthExceededException: bytes can be at most 32766 in length; got 32767
  addDocument(c): threw MaxBytesLengthExceededException: bytes can be at most 32766 in length; got 32767
  getReader: threw IllegalStateException: Can't close queue unless all changes are applied
  flush: ok
  commit: ok
  isOpen=true tragic=null hasUncommittedChanges=true
    committed numDocs=1

• No precondition. deleteDocuments(Term...): "Deletes the document(s) containing any of the terms", documenting only CorruptIndexException and IOException (IndexWriter.java:1855-1863); updateDocument likewise (:1901-1911). Term's constructors call only a null field or null bytes "undefined behavior" (Term.java:47-48).
• Where the limit is documented, the rejection is contained. MAX_TERM_LENGTH: "If a term arrives from the analyzer longer than this length, an IllegalArgumentException is thrown" (IndexWriter.java:252-258); addDocument: "each term in the document can be no longer than MAX_TERM_LENGTH in bytes, otherwise an IllegalArgumentException will be thrown" (:1480-1481). TestIndexWriter#testWickedLongTerm, after that rejection: "Make sure we can add another normal document" (TestIndexWriter.java:1548-1565).
• The writer's failure contract. "if you hit an Error, or disaster strikes during a checkpoint then IndexWriter will close itself" (IndexWriter.java:162-165). A RuntimeException from one call is neither, yet here it makes later, unrelated calls fail. commit(): "Commits all pending changes (added and deleted documents, ...)" (:4189).
• Consistency. The same term as deleteDocuments(new TermQuery(term)) is a no-op on main, and the same call is a no-op on 9.7.0 and 9.8.0 (all executed; earlier releases use the same HashMap buffer).
• History. #12573 ("Use radix sort to speed up the sorting of deleted terms") is listed under 9.9.0 "Optimizations"; its description covers sort time and memory, and its review never mentions term length. We found no test in Lucene's test sources that deletes or updates by a term over 32,766 bytes (searched for MAX_TERM_LENGTH, BYTE_BLOCK_SIZE and MaxBytesLengthExceededException); TestBufferedUpdates#testDeletedTerms, added by #12573, uses keys of 1-3 bytes.

The strongest case: 32,766 bytes is "the absolute hard maximum length for a term" and Lucene's SECURITY.md makes length limits on untrusted input the application's job, so a 32,767-byte delete term is garbage in. And continuing after a failed commit can lose documents, with rollback() as the remedy (as Simon Willnauer said on LUCENE-8692, #9738), so the later loss is not a bug of its own. Lucene also documents no general promise that a call that throws leaves the writer unchanged.

Why it fails: no delete or update API states the limit, and where the limit applies Lucene rejects the call cleanly and keeps the writer usable. Here the bad call is itself rejected, yet unrelated later calls (a normal addDocument, a 3-byte delete) fail too, which breaks failure atomicity rather than being garbage in, garbage out. The same term through TermQuery, and through this API up to 9.8.0, is harmless, and the change came from a performance PR that never considered term length. The LUCENE-8692 point covers only the secondary symptom. Without a general failure-atomicity promise, the specific contracts still apply: the add path shows what Lucene does at the same limit. Verdict: not intended.

Options B and A were built and run on 645d4fe14a20; C is an untested, more general alternative.

In DocumentsWriterDeleteQueue, check every term in addDelete(Term...) and in newNode(Term) (which builds the delete entry of all updateDocument(s)(Term, ...) overloads) before the entry is created, so the call fails on its own and nothing is queued. The check could equally sit in the four public IndexWriter methods; this class is the one place they all pass through.

private static void checkDeleteTerm(Term term) {
  if (term.bytes().length > IndexWriter.MAX_TERM_LENGTH) {
    throw new IllegalArgumentException(
        "delete term in field=\"" + term.field() + "\" is longer than "
            + IndexWriter.MAX_TERM_LENGTH + " bytes: " + term.bytes().length);
  }
}
// called first in addDelete(Term... terms), for each term, and in newNode(Term term)

Trade-offs

• Same exception type addDocument uses for an immense term; the writer stays usable, as testWickedLongTerm expects on the add path.
• A behaviour change against 9.8.0, which accepted such a delete silently (9.9.0 and later already throw, just with the wrong consequences); callers that catch MaxBytesLengthExceededException see IllegalArgumentException instead.
• updateDocument(longTerm, doc) now rejects the whole update, document included, which matches the update's atomic contract. The javadoc of the delete and update methods should then state the limit.

What the checks validated

• Scratch worktree of 645d4fe14a20, JDK 25: the minimal test fails on stock through gradle with assertions on and off, and passes with the fix, assertions on and off.
• Gradle :lucene:core:test with the fix, assertions on: the minimal test plus TestIndexWriterDelete, TestBufferedUpdates, TestDocumentsWriterDeleteQueue, TestIndexWriter, TestIndexWriterExceptions and TestExceedMaxTermLength: 197 tests in 7 suites, 10 skipped, 0 failures.
• checkGoogleJavaFormat reports no violation in the changed file.
• An independent fact check re-applied the diff and re-ran the minimal test (assertions off) and the 7 suites: pass, 197 tests, 10 skipped.
• Not run: the full core suite, other modules, nightly tests, ecjLint and forbiddenApis.

In BufferedUpdates.addTerm, return early when term.bytes().length > ByteBlockPool.BYTE_BLOCK_SIZE - 2: such a delete can match no indexed term (+6 lines, with a comment).

public void addTerm(Term term, int docIDUpto) {
  if (term.bytes().length > ByteBlockPool.BYTE_BLOCK_SIZE - 2) {
    // No indexed term can be longer than IndexWriter.MAX_TERM_LENGTH, so this delete matches
    // nothing; skip it instead of failing in the BytesRefHash, which would leave the node in
    // the shared delete queue and fail every later apply.
    return;
  }
  ...

Trade-offs

• Restores the 9.8.0 outcome (the delete is a no-op), at the place the limit lives, for every path that buffers a delete term; consistent with deleteDocuments(new TermQuery(term)) on main.
• Silent: a caller passing a bogus key gets no signal.
• Not the 9.8.0 state: 9.8.0 put the term in its map, counted it and charged its bytes to the RAM accounting; this fix returns before all of that.

What the checks validated

• The same 7 suites with the fix, assertions on: 197 tests, 10 skipped, 0 failures.
• The minimal test passes with the fix with assertions on and off (standalone runner; also in an independent reviewer's check).
• Not run: the full core suite, nightly tests, the format check.

Drop or skip an entry whose apply throws instead of keeping it for every later apply. This would also cover other ways an entry can throw, such as the delete-buffer overflow addressed by open PR #16867.

Trade-offs

Not built. The entry is shared by the global slice and every per-thread slice, a multi-term entry may have applied some terms before it throws, and simply resetting the slice on a throw would silently drop the deletes of the entries after it. It needs a design discussion; A or B fixes this trigger without it.

Method: code reading, then execution. A hunt unit on maximum-length terms was working through the operation "deleteDocuments(Term...) on long terms" and noticed that since #12573 the delete-term buffer is a BytesRefHash with a 32,766-byte key limit, applied only after the delete is enqueued.

• A plain step probe (add, commit, add, over-long delete, add, NRT reopen, flush, commit, close) on main, the 10.6.0 release commit (then its RC1), 10.5.2, 9.9.0 and 9.8.0 showed the later failures and the lost buffered documents, with 9.8.0 clean; two more probes covered close() right after the delete and the update APIs.
• The sequence was then reduced to the one-method JUnit; the history check ran it, in an earlier 50-line version, on every release from 9.7.0 to 10.5.2 and on 9.8.0 with only #12573's change applied; the fact check ran the final test on the 10.6.0 release.