A document rejected for its custom term frequency closes the IndexWriter and rolls back uncommitted changes

Lucene lets analysis set a per-token term frequency through TermFrequencyAttribute, for example with DelimitedTermFrequencyTokenFilter, which turns red|2 into the term red with frequency 2. When the field also stores term-vector positions or offsets, indexes postings positions (any default TextField), or is DOCS-only and the term repeats, addDocument rejects the document, as upstream tests assert. But the four checks run inside termsHashPerField.add, which IndexingChain treats as aborting, so the rejection also becomes the writer's tragic exception: the IndexWriter closes itself, every add, update and delete since the last commit is rolled back, and every later indexing change or commit throws AlreadyClosedException until a new writer is opened. The neighbouring offset and position checks, and the duplicate-term rejection #15584 added for the same feature in the same try, only reject the document. The bug came in with the feature (LUCENE-7854, 7.0.0) and is in every release up to 10.6.0 and on main.

1. The token loop. IndexingChain.invertTokenStream handles one token at a time. Its position, offset and length checks (IndexingChain.java:1996-2064) throw plain IllegalArgumentExceptions outside the aborting try, so they only reject the document (the comment at :1989-1994 says the same of analyzer exceptions).
2. The aborting call. IndexingChain.java:2068-2073: "If we hit an exception in here, we abort all buffered documents since the last flush, on the likelihood that the internal state of the terms hash is now corrupt". Then termsHashPerField.add(...) (:2074-2075). Only MaxBytesLengthExceededException (:2076-2094) and #15584's DuplicateTermException (:2095-2097) are converted to non-aborting IllegalArgumentExceptions; everything else reaches catch (Throwable th) { onAbortingException(th); throw th; } (:2098-2101).
3. Where the checks are. TermsHashPerField.add (TermsHashPerField.java:190-209) adds the term to the hash, then calls newTerm or addTerm of the postings writer and then of the term-vector writer. The four checks sit inside those calls; three of them run after state was changed, the DOCS-only one before anything is written:
   • postings positions: FreqProxTermsWriterPerField.getTermFreq (:208-219, IllegalStateException "cannot index positions while using custom TermFrequencyAttribute"), called from newTerm (:125) and addTerm (:177, :196);
   • DOCS-only: FreqProxTermsWriterPerField.addTerm (:145-152, IllegalStateException "must index term freq while using custom TermFrequencyAttribute"); it runs only when the term is already in the in-memory segment, because newTerm has no such check (:119-122), and it throws at the top of addTerm, before any write (for an existing term bytesHash.add only returns its id);
   • term-vector positions and offsets: TermVectorsConsumerPerField.getTermFreq (:286-307, IllegalArgumentException), called from newTerm (:270) and addTerm (:281).
4. Abort and close. DocumentsWriterPerThread.maybeAbort (:594-605, called at :310) aborts the in-memory segment and fires the tragic event; IndexWriter.updateDocuments (IndexWriter.java:1625-1642, used by addDocument) then calls maybeCloseOnTragicEvent (:1639), which rolls the writer back (:6098-6106).

The study's four-case test commits doc 1, buffers doc 2, adds the rejected document, then adds doc 3 and commits. Expected for every case: tragic=null isOpen=true nextAddAndCommit=ok committedDocs=3. Actual on main, for term-vector positions, term-vector offsets, postings positions and DOCS-only: tragic=<the rejection> isOpen=false nextAddAndCommit=AlreadyClosedException committedDocs=1. Its controls pass: backwards offsets on the same field type keep the writer open with 3 documents, and frequency 3 on a DOCS_AND_FREQS field without term vectors is indexed and reads back 3.

With the shipped filter on a field with term-vector positions, 5 committed and 5 buffered documents, one document red|2 green makes the rejection tragic. On a default TextField, documents without a delimiter index normally for the whole session; the first words|2 closes the writer and the 5 buffered documents are lost.

• Who. Users of the opt-in custom-term-frequency feature, through their own TokenStream or the shipped DelimitedTermFrequencyTokenFilter (factory name delimitedTermFrequency), on a field that stores term-vector positions or offsets (allowed by TermFrequencyAttribute's stated precondition; the filter's javadoc, "no positions or offsets", may be read as ruling these out too), indexes postings positions (any default TextField; the filter's javadoc forbids it) or is DOCS-only (the attribute's javadoc asks for DOCS_AND_FREQS).
• What it costs. One such document closes the writer. Every add, update and delete since the last commit is rolled back: in the study's probe a document deleted after the last commit was live again in the reopened index, and an update was undone. Other threads' buffered documents go too; that follows from the rollback but was not executed (all runs were single-threaded). The application must open a new writer and re-index everything since the last commit.
• Not affected. The committed index stays consistent and a new writer works; an NRT reader opened earlier keeps its view. FeatureField, the core field that uses TermFrequencyAttribute, has no positions and no term-vector positions or offsets, so it never reaches these checks (code reading).
• Conditions. Default IndexWriterConfig with no codec configured (the gradle runs used the test framework's randomized Asserting(Lucene104) codec; the core-only probe runs with Lucene's default codec), assertions on and off, ordinary scale, one thread. Data-dependent: with the filter, the first word|N with N != 1 fires it.
• How likely: rare. It needs the opt-in feature plus a field setting the javadoc rules out or an unusual term-vector choice, and the failure is loud; but a misconfigured field can run for a long time before the first such token.

The field meets TermFrequencyAttribute's precondition (DOCS_AND_FREQS). The token comes from the test framework's CannedTokenStream with one Token whose term frequency is 2; the writer uses the default IndexWriterConfig. The test sits in org.apache.lucene.index, as Lucene's own tests do, but uses no package-private API. Line 40 of the file (expectThrows, counting the license header) passes, so the documented rejection is unchanged; only line 42 fails. No commit or extra documents are needed: the tragic exception alone shows the writer closed, and the rollback of uncommitted work follows from it (shown by the study's longer test, in the collapsed explanation above).

The other three cases (term-vector offsets, postings positions, DOCS-only) share the root cause and were executed by the study's longer test; they are not part of this repro.

Regression test: this same test; it passes with the recommended fix.

cp $REPRO/TestCustomTermFreqRejectionKeepsWriterOpen.java $LUCENE/lucene/core/src/test/org/apache/lucene/index/
cd $LUCENE && ./gradlew :lucene:core:test --tests TestCustomTermFreqRejectionKeepsWriterOpen -Ptests.seed=DEADBEEF -Ptests.asserts=true
# assertions off fails the same way: -Ptests.asserts=false

# gradle, stock 645d4fe14a20, -Ptests.seed=DEADBEEF -Ptests.asserts=true (stack trimmed; -Ptests.asserts=false identical)
TestCustomTermFreqRejectionKeepsWriterOpen > testTermVectorPositionsRejectionKeepsWriterOpen FAILED
    java.lang.AssertionError: tragic exception expected null, but was:<java.lang.IllegalArgumentException: field "f": cannot index term vector positions while using custom TermFrequencyAttribute>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:EED294BF6AE05155]:0)
        at org.junit.Assert.assertNull(Assert.java:738)
        at org.apache.lucene.index.TestCustomTermFreqRejectionKeepsWriterOpen.testTermVectorPositionsRejectionKeepsWriterOpen(TestCustomTermFreqRejectionKeepsWriterOpen.java:42)
> Task :lucene:core:test FAILED

# the study's four-case test on the same stock build (gradle, seed DEADBEEF): 6 tests completed, 4 failed
testTermVectorPositions  expected:<tragic=[null isOpen=true nextAddAndCommit=ok committedDocs=3]> but was:<tragic=[IllegalArgumentException isOpen=false nextAddAndCommit=AlreadyClosedException committedDocs=1]>
testTermVectorOffsets    same, IllegalArgumentException
testPostingsPositions    same, IllegalStateException
testExtraDocsOnlyRepeatedTerm  same, IllegalStateException

• Same feature, same try, opposite outcomes. #15584 (10.5.0) added a custom-frequency rejection (a duplicate term on a DOCS_AND_CUSTOM_FREQS field) in this very try and made it non-aborting (IndexingChain.java:2095-2097); TestCustomTermFreq.java:125 says "ensure IndexWriter remains open and doc was skipped". The PR's author, msokolov, wrote in the PR conversation that they "found that the exception I raised for duplicate terms in termdoc fields would cause the writer to crash, so added some exception handling that merely skips the doc being indexed (same as for a too-large term)". With the same index options (DOCS_AND_CUSTOM_FREQS; the duplicate-term case without term vectors), the duplicate term keeps the writer open with 3 documents committed, while frequency 2 with term-vector positions closes it with 1.
• The precondition is met. TermFrequencyAttribute asks only for DOCS_AND_FREQS (TermFrequencyAttribute.java:23-27); the DOCS_AND_CUSTOM_FREQS javadoc says nothing about term vectors (IndexOptions.java:52-56). Upstream's own test of the term-vector rejection uses DOCS_AND_CUSTOM_FREQS + term-vector positions (TestCustomTermFreq.java:471-496).
• Nothing documents the close. The IndexWriter javadoc names "an Error, or disaster strikes during a checkpoint" (IndexWriter.java:162-165); addDocument says after an exception "the index will be consistent, but this document may not have been added" (:1465-1466); LUCENE-5987 describes aborting exceptions as ones "that would otherwise cause silent data loss".
• Precedent. LUCENE-5662 is listed under "Bug fixes" in 4.8.1; testNullStoredField and testCrazyPositionIncrementGap assert getTragicException() == null after a rejected document, and LUCENE-5611's testNoAbortOnBadTVSettings checks "the exc didn't lose our first document".
• Why upstream tests never noticed. The four rejection tests end with IOUtils.close(w, dir) and never look at the writer, and close() after a tragic close does nothing (close(), IndexWriter.java:1363-1369, calls shutdown() or rollback(), which stop at shouldClose, :1374-1392). The LUCENE-7854 discussion (34 comments) never mentions aborting.

The strongest case: this is misuse, because TermFrequencyAttribute requires DOCS_AND_FREQS and DelimitedTermFrequencyTokenFilter says "no positions or offsets"; and every exception from add is aborting on purpose, because the terms hash may be corrupt. The study confirmed that part for the postings-positions check: a shadow that only skips the abort, run on a default TextField, corrupts the in-memory postings (AssertionError with assertions on; docs out of order or cannot write negative vLong at flush with assertions off). It was not run for the term-vector or DOCS-only checks; the DOCS-only check throws before anything is written.

Why it fails: for TermFrequencyAttribute (the repro's API) the misuse argument covers only the postings-positions and DOCS-only cases (it lowers their severity); the term-vector cases are inside the attribute's stated precondition. The filter's javadoc can be read as also ruling out term-vector positions and offsets. The second half explains why the abort is needed once a check fires mid-write, not why a deterministic check of the token against the field type must run there: moving it before add keeps the exception types and messages, keeps the writer open and passes upstream's tests. A maintainer may still label the change an improvement.

Option A was built and run. B, a simpler variant, was run too and breaks an upstream test. C is the study's earlier, partial fix. D is not recommended.

In IndexingChain.invertTokenStream, right after the offset checks (IndexingChain.java:2051), a token whose frequency is not 1 goes through a new validateCustomTermFrequency(field). In the order the old checks fire, it throws IllegalStateException "must index term freq ..." for a DOCS-only field when the term is already buffered (a new TermsHashPerField.containsTerm, bytesHash.find(termBytes) >= 0, true exactly when add would call addTerm), IllegalStateException "cannot index positions ..." when postings have positions, and IllegalArgumentException "cannot index term vector positions / offsets ..." when the field stores term vectors with positions or offsets.

if (invertState.termFreqAttribute.getTermFrequency() != 1) {
  // Check here, not inside termsHashPerField.add below: an exception thrown from there
  // aborts all buffered documents, while this one only rejects the current document.
  validateCustomTermFrequency(field);
}

Trade-offs

• Old checks. The checks inside add stay; with the fix they are no longer reached from IndexingChain. They could become assertions.
• Which exception wins. A document with two problems may report a different one first (for example the term-vector check before #15584's duplicate-term check; both reject only the document). It also runs before the field-length overflow check (IndexingChain.java:2053-2064): a term repeated on a DOCS-only field whose frequency would also overflow the field length gets the DOCS-only IllegalStateException instead of "too many tokens"; both reject only the document.
• Cost. One int comparison per token; for ordinary fields the frequency is already read for the field length (IndexingChain.java:2057-2059), for DOCS_AND_CUSTOM_FREQS fields it is one extra attribute read. The field checks, and for DOCS-only fields one hash lookup, run only for frequencies other than 1. No benchmark was run.
• Conflicts. Open PR #16269 ("Lazily build the term-vectors per-field", states no behaviour change) also edits IndexingChain.java and TermsHashPerField.java, in other hunks; it leaves the checks and the abort handling unchanged, so at most a textual rebase is needed.

What the checks validated

Applied to a scratch worktree of 645d4fe14a20 and run through gradle with OpenJDK 25.0.3, seed DEADBEEF (2026-10-30):

• stock: the minimal test fails with assertions on and off; the study's four-case test fails 4 of 6;
• with the fix: the minimal test passes with assertions on and off; the four-case test passes 6 of 6 with assertions on and off (all four cases keep the writer open with 3 committed documents);
• the closest suites TestCustomTermFreq, TestTermVectorsWriter, TestTermVectors, TestIndexWriterExceptions and TestFeatureField pass (88 tests, 0 failures, 3 skipped), and so does TestDelimitedTermFrequencyTokenFilter (3 tests);
• the whole :lucene:core:test passes (748 classes, 8,897 tests, 0 failures, 322 skipped; run after the two copied tests were removed, so it checks upstream core with the patch); checkGoogleJavaFormat, ecjLintMain and forbiddenApisMain for core pass.
• assertions on and off were run for the minimal and the four-case test; the closest suites, the filter's tests and the whole core module ran with assertions on.

The same check without the hash lookup (+42 lines, IndexingChain.java only). It matches the attribute's javadoc and the comment "you can't index DOCS_ONLY with custom term freq" (TestCustomTermFreq.java:373).

Trade-offs

Changes behaviour an upstream test relies on: the minimal test, the four-case test and the filter's tests pass, but TestCustomTermFreq.testOverflowInt, which indexes frequencies 3 and Integer.MAX_VALUE on a DOCS field and expects the length-overflow IllegalArgumentException, now gets IllegalStateException: ... must index term freq .... It was the only failure in the org.apache.lucene.index package (1402 tests). Its lint run is incomplete: checkGoogleJavaFormat failed on a formatting nit in the copied four-case test, not in the patch. A stricter DOCS-only rule would be a separate change.

Moves the postings-positions and term-vector checks before add, not the DOCS-only check.

Trade-offs

The repro passes, and so do upstream TestCustomTermFreq (15 tests) and the term-vector, exception and addIndexes suites the study ran (56 and 92 tests); but a DOCS-only field with a repeated term still closes the writer (the four-case test fails 1 of 6).

Convert them in the catch chain, as #15584 did for DuplicateTermException.

Trade-offs

Unsafe for the postings-positions check: it fires after the postings were partly written, and the study's shadow that skips the abort corrupts the in-memory postings (AssertionError with assertions on; CorruptIndexException: docs out of order or cannot write negative vLong at flush with assertions off). The term-vector checks also fire after state was written (not tested with a shadow), and moving them to the top of newTerm is not enough, because for a new term the hash and stream slices were already updated. The DOCS-only check throws before anything is written, so a targeted non-aborting catch might be safe there, but that was not tested; checking all four before add avoids the question.

Method: code reading, then executed public-API tests (the hunt classes it as non-formal). The finder was building a Z3 (SMT) model of the term-vector encoding; the model covers the encoding, not this exception path, and no formal tool witnessed the bug.

• First test. Commit one document, buffer one, add a document whose token stream sets TermFrequencyAttribute to 3, then add and commit another: all three of its bug cases closed the writer, and its control (offsets going backwards, rejected before add) kept it open.
• Widening. An independent reproducer, writing its own test from the claim only, found the DOCS-only case; a devil's-advocate verifier showed the data dependence with the shipped filter and that a non-aborting catch would corrupt the postings of a default TextField (postings with positions).
• The minimal repro above is the term-vector-positions case cut down to one token and one assertion.