A Monitor.register call that IndexWriter rejects is not rolled back: the rejected query and the queries after it in the call are deleted at the next write

Monitor.register updates stored queries by id. When IndexWriter rejects one of the new query documents without aborting, for example because a TermFilteredPresearcher filter-field value is longer than 32,766 bytes, register() throws IllegalArgumentException, but WritableQueryIndex.commitWithoutNotify has already queued the delete of every query id in the chunk and, for each entry up to and including the rejected one, put the new version into its in-memory cache (and added the new documents before it); nothing is rolled back. Right after the failure the rejected query matches nothing while getQuery still returns its old version, and the queries after it still match their old versions. The next successful register or deleteById commits the queued deletes, and so does close() (default commitOnClose): the rejected query and the existing queries after it in the call are deleted, and the queries before it take their new version although the call threw. So a persistent Monitor can reopen without valid queries of the same call. With the default configuration a new query with a 32,765-byte id does the same to the queries after it. A call that fails earlier, while its documents are built, loses nothing. Two queries suffice, assertions on or off, and every release that ships the monitor module, 8.2.0 to 10.6.0 (10.6.0 by reading), and main are affected.

• Monitor (lucene/monitor): stores queries under ids and reports which stored queries a document matches. It is backed by its own small Lucene index, the query index.
• Two copies of each query. Each query is split into parts by the QueryDecomposer (for example one per clause of a plain disjunction; it keeps a BooleanQuery whole when minimumShouldMatch > 1, when it has several mandatory clauses, or mandatory and optional clauses together, QueryDecomposer.java:77-100; a term query is one part). Each part becomes one query-index document (the terms the presearcher selects on, the filter-field metadata, the serialized query, which is empty without a serializer, WritableQueryIndex.java:192, :198) and one cache entry named id + "_" + part (QueryCacheEntry.java:56), so q2's only part is q2_0, in the old and in the new version.
• Matching: the document's terms select candidate query-index documents; for each candidate the Monitor looks up its cache name and runs that cached query (WritableQueryIndex.java:380-386). getQuery and getQueryIds read the query index (QueryIndex.java:62-72, Monitor.java:276-280).
• Filter fields: TermFilteredPresearcher(weightor, handlers, Set.of("lang")) checks a document only against queries with the same lang metadata; the value is indexed as one term (TermFilteredPresearcher.java:244-252, KeywordAnalyzer at MonitorConfiguration.java:44). IndexWriter rejects terms over 32,766 bytes (IndexWriter.java:258, :1480-1481).

1. Monitor.register commits its input in chunks of up to queryUpdateBufferSize + 1 = 5001 entries (Monitor.java:152-162, MonitorConfiguration.java:35; one more than the setter's javadoc describes, a harmless off-by-one).
2. WritableQueryIndex.commitWithoutNotify builds every document first (buildIndexables, :185-202); serializer, decomposer and presearcher errors are thrown here, before any write, and lose nothing.
3. It then queues writer.deleteDocuments(Term(_query_id, id)) for every id of the chunk (:117-119), and for each part puts the new cache entry (:121) before writer.addDocument (:122). There is no catch and no rollback: the only try (:111-131) is a finally that releases purgeLock, so an exception skips writer.commit() and maybeRefresh() (:127-128). The add loop runs in input order and stops at the rejected entry.
4. IndexWriter rejects the document cleanly ("Document contains at least one immense term in field=\"lang\"") and stays usable, but the queued deletes and the documents added before the failure stay buffered.
5. Right after the failure the searcher still shows the old documents and getQuery the old versions; what match() runs depends on the position in the chunk. The rejected entry and the entries before it have their new cache entries: the old q2_0 entry (selected by "b") now runs y, so q2 matches nothing, and a valid update before it matches neither version in the probes (it can also over-match, see the Downstream note). Entries after the rejected one are untouched and still match their old version until the next write. purgeCache() does not repair it (it rebuilds the cache from the current cache, :249-256).
6. The next write (register or deleteById, :338-344) commits the buffered changes: queries before the rejected one switch to their new version; the rejected query and the existing queries after it lose both versions (a new id after it is never added). That holds for one-part queries, as in every executed case; by code reading, a multi-part query whose later part is rejected keeps the new parts added before it. close() closes the writer, whose config keeps the default commitOnClose = true (MonitorConfiguration.java:43-50, IndexWriterConfig.java:102), so it commits them too.

With the valid update before the rejected one, register(q1 = c, q2 rejected) over q1 = a, q2 = b: live, getQuery("q1") = a but q1 matches neither "a" nor "c"; after close and reopen q1 = c and q2 is gone. With the valid update after the rejected one (the default-configuration row), q1 still matches "a" right after the failure and is deleted by the next register. I/O variants: when the query index cannot create a file during register(q = b) over q = a (simulated full disk), q matches neither "a" nor "b", and the writer is closed, so the next register fails with AlreadyClosedException; in the failure-injection matrix (other queries) the same failure leaves getQuery at the old versions, and a reopened Monitor has them. A simulated sync failure at commit leaves the writer usable, and the next register commits the update.

• Triggers (executed): a TermFilteredPresearcher filter-field metadata value longer than 32,766 UTF-8 bytes (filter fields are opt-in); with the default configuration, a query id whose cache id <id>_<part> is over the 32,766-byte doc-values limit while the id itself is at most 32,766 bytes, in the same call as other updates (contrived): for a one-part query an id of 32,765-32,766 bytes (32,764 passes); in general UTF8(id).length + 1 + digits(part) > 32766 (by code reading). An id of 32,767 bytes or more fails earlier, in writer.deleteDocuments, and leaves its term in the writer's delete queue, so later writes and close() throw: a separate root cause. An I/O error while the documents are written (simulated createOutput failure): in-session wrong matches; the writer then closes and later writes fail loudly. A simulated sync failure at commit leaves the writer usable, and the next write commits the update. By construction also any custom Presearcher whose document IndexWriter rejects, for example one that copies an oversized query term into the document (not run).
• Not triggers (executed with the default TermFilteredPresearcher): an oversized term inside the query itself, a null query string, and serializer, decomposer or presearcher exceptions. They fail while the documents are built, before any write, and lose nothing.
• Blast radius: every query id of the failed chunk, up to 5,001 with the default queryUpdateBufferSize, gets a queued delete. The rejected query and the existing queries after it are lost; the queries before it take their new version although the call threw.
• Default vs opt-in: the filter-field path needs the opt-in filter fields; the long-id path needs only new Monitor(analyzer). In-memory Monitors lose the queries at the next write; persistent Monitors also at close(). Assertions make no difference.
• Scale: none; two queries suffice.
• Damage: silent loss of stored queries (alerts stop firing): the rejected query at once, the queries after it at the next write, then permanently; only the first call throws, and the deleting write that follows succeeds without error. Until then getQuery and match() disagree for the rejected query and the updates before it. One in-session over-match was found (see the Downstream note). Not observed in the probes: structural index corruption, a hang, or an effect on ordinary IndexSearcher searches.
• How likely: rare (a judgement): it needs a value or id over 32 KB, or an I/O error, during a register; no report was found in the searches below, although the code dates from 8.2.0 (2019).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<[q1]> but was:<[]>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:120)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.monitor.TestFailedRegisterDeletesOtherQuery.testRejectedQueryDoesNotDeleteOtherQueryOfTheCall(TestFailedRegisterDeletesOtherQuery.java:44)

• Contracts. register: "Add new queries to the monitor" (Monitor.java:147-151, :168-173), with no rollback or partial-failure semantics documented; getQuery: "@return the MonitorQuery stored for this id" (Monitor.java:245-255); match calls the matcher "for each possible matching query" (Monitor.java:207-209): after the failure q2 is stored as b, b matches the document, and match does not report it. MonitorQuery metadata: "Must not have any null values" (MonitorQuery.java:40-41), no length limit.
• IndexWriter: after an exception in addDocument "the index will be consistent, but this document may not have been added" (IndexWriter.java:1465-1466); updateDocuments "Atomically deletes ... and adds a block of documents ... such that an external reader will see all or none of the documents" (:1594-1596). On the same rejection updateDocuments keeps the old version; a separate delete plus a failing add loses it (executed).
• Tests. TestMonitorPersistence#testCacheIsRepopulated (TestMonitorPersistence.java:54-87) registers 4 queries, has a fifth register throw (a null query string, rejected before any write), reopens and asserts that all 4 are still there and still match. That is consistent with this report's expectation but shows it only for a failure before any write. No upstream test injects a failure into the write loop (TestMonitorErrorHandling covers match-time errors and the MonitorQuery null-metadata check).
• History. The delete-every-id-first loop came with the module (LUCENE-8766, 2019, code donated from Luwak) and moved unchanged in LUCENE-10422 (#679); git log -S updateDocuments and -S rollback on the module find nothing. In Luwak the loop was created by the fix of luwak#68 (2015, a concurrency race); before it each query was one atomic updateDocument (luwak#21). No discussion of a failing add was found. Context: in #11606 (2022) @hossman notes that the Monitor API works "to add+commit an entire batch all at once", so large batches are normal.

The strongest case for "intended or misuse" (built by the devil's-advocate verifier; the GPT-5.6 reviews rated it moderate): register promises no atomicity and commits in chunks; filter fields are documented as "keyword fields" (package-info.java:65), so a 40,000-byte value is garbage in, and IndexWriter documents its 32,766-byte term limit; after the simulated createOutput IOException the writer is closed anyway.

Why it fails: it defeats only whole-call atomicity and the I/O variant. It does not explain why a rejected update later deletes the query's old version, why it deletes other, valid queries of the call, or why getQuery and match() disagree. The input is rejected only after the Monitor has acted on it; if oversized values are illegal, the fix is to reject them before the first delete. The same call failing a few lines earlier loses nothing, and nothing documents that the point of failure decides whether queries survive. The deletion is deferred to a later, successful, unrelated call. No won't-fix or by-design discussion was found. The case is strongest for the I/O variant: after a tragic createOutput failure the writer is closed and a re-created Monitor has the committed old versions (a GPT-5.6 review rated that case strong, the filter-field and long-id cases moderate); a sync failure at commit is not tragic. The claim rests on the non-aborting rejections. Verdict: a defect, claimed per query, not per call.

Two fixes were built and validated in a scratch worktree at ea57360912ed (removed afterwards), plus an optional add-on for a failing commit. Workaround (partial): length-check filter-field values and ids before register (filter-field values at most 32,766 UTF-8 bytes; ids short enough that <id>_<part> stays within 32,766 bytes) and do not batch untrusted registrations of different users into one call; after a register that throws, register again, before any other write, what getQuery() returns for each id of the call (needs a serializer; executed on main and 10.5.2 for the filter-field case, with ids that existed before the call). It does not undo a new id placed before the rejected entry, whose buffered document the next write commits, and before 9.9.0 getQuery on an absent id can throw NullPointerException (GITHUB#12736).

Replace the per-id deletes and per-document adds with one IndexWriter.updateDocuments(Query, Iterable) call, the query form of the all-or-none update, and put the cache entries after it. A chunk IndexWriter rejects then changes nothing, exactly like a chunk that fails today while its documents are built, so callers get one rule. This is one policy choice (register documents no atomicity); the per-id option keeps more of a failed chunk. The cache is still updated before maybeRefresh(), and only commitWithoutNotify refreshes the writable searcher, under commitLock. Diffs: option-a.diff, option-a-with-test.diff (both apply cleanly at ea57360912ed).

if (indexables.size() > 0) {
  Set<BytesRef> ids = new HashSet<>();
  List<Document> docs = new ArrayList<>(indexables.size());
  for (Indexable update : indexables) {
    ids.add(new BytesRef(update.queryCacheEntry.queryId));
    docs.add(update.document);
  }
  // one atomic update: if IndexWriter rejects any document, none of the deletes and none
  // of the documents is applied, and the cache below is left unchanged
  writer.updateDocuments(new TermInSetQuery(FIELDS.query_id, ids), docs);
  for (Indexable update : indexables) {
    this.queries.put(update.queryCacheEntry.cacheId, update.queryCacheEntry);
    if (purgeCache != null)
      purgeCache.put(update.queryCacheEntry.cacheId, update.queryCacheEntry);

Trade-offs

• All or none per chunk: one bad entry also blocks the valid entries of its chunk, which keep their old versions (executed). That is today's behaviour for build-time failures. Earlier chunks of a register(Iterable) with more than 5,001 entries are still committed before a later chunk fails (unchanged).
• A chunk is added as one document block, kept in one segment, and the delete becomes one delete-by-query instead of one delete-by-term per id. Memory and speed were not measured. Both updateDocuments overloads are @lucene.experimental.
• Not covered: a failing commit() (for example fsync) after a successful update leaves the cache ahead of the searcher until the next successful commit (the remaining matrix case); see the add-on.
• Unchanged: two versions of one id in one call (LFV-332) still keep both; that fix, in buildIndexables, composes with this one. Queries already deleted by an affected version stay deleted.
• Not tested: LFV-332's case together with this one, a concurrent same-id deleteById or clear, an active purge (the purge-cache handling is kept, by code reading).
• Not reviewed by a committer.

What the checks validated

• Stock: this doc's test fails 4 of 5 with assertions on and off; with the fix it passes 5 of 5 (gradle and the harness).
• The whole monitor module with the fix: 33 classes, 186 tests (181 upstream plus the 5 new ones), 0 failures, 1 skipped, assertions on and off.
• :lucene:monitor:check -x test passes (google-java-format, ecjLint, forbiddenApis); tidy leaves the patch unchanged.
• With the fix shadowed on the main jars: the finder's two classes (4 tests, including the I/O test) and the independent reproducer's (6 tests) pass; a probe shows that updates still replace the old version, in memory and after a reopen, and that deleteById works; the fail-open probe passes; the failure-injection matrix drops from 2 to 1 bad case of 9.

On top of the recommended fix, wrap writer.commit(): if it throws, call manager.maybeRefresh() (suppressed into the exception) before rethrowing (option-a-refresh.diff is a diff against stock that already contains the recommended fix: +20/-7 in all, +12/-1 beyond it; apply it instead of option-a.diff, not on top of it).

Trade-offs

A register that threw at commit has its update visible in-session, and the next successful commit persists it; callers who treat any exception as "nothing happened" may be surprised. It catches Throwable, so it also refreshes after an Error or a tragic failure; that error-path policy needs review. A separate concern from the rejected-document bug.

What the checks validated

The failure-injection matrix has 0 bad cases of 9 (the recommended fix alone: 1); the whole monitor module passes (186 tests, assertions on).

Group the chunk's documents by id, call writer.updateDocuments(new Term(FIELDS.query_id, id), parts) per id and put that id's cache entries after it; on an exception commit and refresh what was applied, then rethrow (option-b.diff, +26/-10; it catches Throwable, so it also commits after an Error or a tragic failure, which needs review).

Trade-offs

The call is applied partly: ids before the rejected one take their new version, the rejected one and those after it keep the old one (executed), so callers must check which ids took effect. More code, with a commit in the error path. By code reading, if the failure closes the writer (an I/O error), ids applied earlier in the chunk keep their new cache entries while the searcher can no longer be refreshed (not executed).

What the checks validated

This doc's test 5 of 5, the finder's 4 and the independent reproducer's 6 pass with it; the whole monitor module passes (186 tests, assertions on); the fail-open probe passes; the matrix has 1 bad case of 9. Tidy re-wrapped one line; the diff is the tidied version.

In buildIndexables, throw IllegalArgumentException when a filter-field value or a cache id is longer than 32,766 UTF-8 bytes.

Trade-offs

Smallest change and a clearer error, but it covers only the known triggers: a custom presearcher's rejected document, other non-aborting IndexWriter rejections and I/O errors still go through the unprotected loop. A good complement to either fix. Not validated.

Method: TLA+ model checking (TLC 1.8.0), in a hunt unit (monitor lifecycle and persistence, 2026-10-26).

• The model (MonitorWrite.tla) covers the write protocol of WritableQueryIndex: parts per query, non-aborting add failures, concurrent deleteById/clear, close and reopen, with a map from each model element to its Java line. In the single-threaded configuration with a failing add (2 ids, up to 3 parts) TLC violated FailureNoOp (after a register that threw, an id's visible documents are the ones it had before) and AtomicQuery (each id's visible documents are one whole version, and its cache entries are those documents' own). The modelled fix (FIX = TRUE, MonitorWrite.tla:29-31: one atomic updateDocuments(Term(id), parts) per query with the cache puts after it, and the deletes of deleteById/clear under the commit lock; close to the per-id option, not the recommended one) holds both, checked with concurrent register, deleteById and clear (MC_fix_all, 13,440 distinct states).
• A failure-injection matrix on the stock jar (9 injection x operation cases) found 2 broken cases, both in register; operation-level model-based tests without failures found nothing.
• Caveat: the hunt's record dates the find shortly before the first preserved TLC run, so the idea may have come while reading the code to write the model; model checking then formalized and reproduced it.
• Then: a refute review by an independent reviewer from another model family (OpenAI GPT-5.6, reading only); an independent reproducer; a GPT-5.6 "works as designed" review; a devil's advocate; an adjudication; a release sweep, a prior-report search, a security screen and an audit; and this doc's own test, fixes and probes.