A repeated non-UTF-8 binary term in a DOCS_AND_CUSTOM_FREQS field can close IndexWriter instead of skipping the document

IndexOptions.DOCS_AND_CUSTOM_FREQS (new in 10.5.0, #15584) stores a custom score per term and allows each term only once per document. When a document repeats a term, #15584 made addDocument reject just that document with the non-aborting IllegalArgumentException("Document update skipped due to duplicate termdoc term"); TestCustomTermFreq.testRepeatedTerms asserts the writer stays open. On the way, TermsHashPerField.add re-throws the internal DuplicateTermException with the term appended through termBytes.utf8ToString() (TermsHashPerField.java:203 at 5adff78bb168). For some binary terms that are not valid UTF-8, such as a BytesRef that holds just the byte 0xC3, that decode throws ArrayIndexOutOfBoundsException. The indexing chain treats DuplicateTermException (like a too-long term) as a per-document rejection and every other exception as aborting, so the writer records the AIOOBE as its tragic exception and closes, and every document added since the last commit is lost. A one-line fix renders the term with Lucene's existing Term.toString(BytesRef), which prints non-UTF-8 terms as bytes.

1. The rejection starts as intended. For a term-doc field, a term already seen in this document throws DuplicateTermException("field 'f' has duplicate term") (index/FreqProxTermsWriterPerField.java:193-195).
2. The message decoration decodes the term. TermsHashPerField.add catches it and re-throws new DuplicateTermException(e.getMessage() + " '" + termBytes.utf8ToString() + "'") (index/TermsHashPerField.java:200-204, the call at :203). This is the fix site.
3. utf8ToString is documented to throw on bad input: "May throw an AssertionError or a RuntimeException if the data is not well-formed UTF-8" (util/BytesRef.java:133-136). For {C3}, a 2-byte lead byte that ends its array, UnicodeUtil.UTF8toUTF16 reads past the end of the 1-byte array (util/UnicodeUtil.java:649).
4. The indexing chain classifies by exception type. Around termsHashPerField.add (index/IndexingChain.java:2075): MaxBytesLengthExceededException becomes a non-aborting IAE that prints the term's bytes with Arrays.toString (:2076-2094); DuplicateTermException becomes the non-aborting IAE (:2095-2097); every other Throwable goes to onAbortingException (:2098-2100, :176-180). That marks the per-thread writer aborted (index/DocumentsWriterPerThread.java:56-60); its maybeAbort drops the buffered documents and reports a tragic event (:594-605), which IndexWriter records (index/IndexWriter.java:6067-6083); the catch in IndexWriter.updateDocuments then calls maybeCloseOnTragicEvent (:1632-1640), which rolls the writer back and closes it (:6097-6104).
5. Reach. Field(String, BytesRef, IndexableFieldType) builds an untokenized binary field (document/Field.java:187-226) that is inverted as a token stream (Field.java:460-461), so it takes this path. A custom TokenStream with binary terms reaches it too (the devil's-advocate verifier's probe).

• Only some non-UTF-8 terms trigger it. With assertions off, the decode throws only when it runs off an array: a truncated multi-byte sequence at the end of the term's backing array, such as [c3] or [41 e2 82]. The decoder reads past BytesRef.length, so the backing array is what counts: a [c3] slice followed by another byte only garbles the message, while a slice that ends in a 4-byte lead byte ([f0] of [f0 90 80 80]) overruns the decoder's output array and closes the writer too.
• A lone continuation byte such as [80 41] gives a garbled message ('\u0080A': an invisible U+0080, then A) and the writer stays open; with assertions on, that one closes the writer too, through an AssertionError.
• An ASCII term (foo, [41 42]) or a valid UTF-8 term ([c3 a9], "é") repeated the same way gets the intended rejection, and the writer stays open.
• Committed data is not affected.
• Not run: the BINARY invert path (IndexingChain.invertTerm, :2127-2170). StringField and KeywordField fix their index options to DOCS, but a custom IndexableField that returns InvertableType.BINARY, or an untokenized column of the columnar API (10.6.0), can use this option. By reading, invertTerm has neither the duplicate-term catch nor the aborting catch, so there the caller would get the raw exception instead of the IAE.

• Who. Code that indexes raw binary terms into a DOCS_AND_CUSTOM_FREQS field (no Lucene field class uses the option) and lets a document repeat one of them, which the option forbids. Only some byte patterns trigger it (see Scope above).
• Default vs opt-in. An opt-in index option; default IndexWriterConfig and codec otherwise.
• Effect. Loud, not silent: addDocument throws, the writer closes, later calls throw AlreadyClosedException until a new writer is opened, and all documents added since the last commit are lost. No corruption was observed: the writer rolls back to the last commit, and committed data stays intact.
• Scale. One document, a one-byte term; fixed input, single thread.
• Likelihood. Rare: it needs the new option, raw binary terms, a repeated term (a user error) and a byte pattern that makes the decoder run past the end.
• Text vs bytes. CharTermAttribute terms are encoded to valid UTF-8 (unpaired surrogates become U+FFFD), so they get the intended rejection; a token stream that emits raw bytes (BytesTermAttribute, "if binary terms should be indexed") can reach the bug.

TestRepeatedBinaryTermDocTerm (one method, extends LuceneTestCase, package org.apache.lucene.index of the lucene/core tests, Apache license header). Its input is fixed (LuceneTestCase still randomizes codec, similarity and locale, so the command pins a seed; an equivalent test also failed with four other seeds) and it passes checkGoogleJavaFormat. It fails the same way with assertions on and off, and passes unchanged with the recommended fix.

cp $REPRO/TestRepeatedBinaryTermDocTerm.java $LUCENE/lucene/core/src/test/org/apache/lucene/index/
cd $LUCENE && ./gradlew :lucene:core:test --tests org.apache.lucene.index.TestRepeatedBinaryTermDocTerm -Ptests.seed=DEADBEEF

# gradle, scratch worktree at 5adff78bb168, -Ptests.seed=DEADBEEF (assertions on)
TestRepeatedBinaryTermDocTerm > testRepeatedNonUtf8TermDoesNotCloseWriter FAILED
    junit.framework.AssertionFailedError: Unexpected exception type, expected IllegalArgumentException but got java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 1
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:48CCA36ED319B27F]:0)
        ...
        Caused by:
        java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 1
            at org.apache.lucene.util.UnicodeUtil.UTF8toUTF16(UnicodeUtil.java:649)
            at org.apache.lucene.util.BytesRef.utf8ToString(BytesRef.java:139)
            at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:203)
            at org.apache.lucene.index.IndexingChain$PerField.invertTokenStream(IndexingChain.java:2075)
            ...
            at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1492)

# same with -Ptests.asserts=false

# control probe (1 committed doc, 1 buffered doc, then the test document; assertions off)
PROBE ASCII 'foo' twice: thrown=java.lang.IllegalArgumentException: Document update skipped due to duplicate termdoc term ... writerOpen=true tragic=null committedNumDocs=2
PROBE binary {C3} once: thrown=none writerOpen=true tragic=null committedNumDocs=3
PROBE binary {C3} twice: thrown=java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 1 writerOpen=false tragic=java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 1 committedNumDocs=1

• Intent of the feature's author. #15584's author, Michael Sokolov, on 2026-03-23 (https://github.com/apache/lucene/pull/15584#issuecomment-4110154752): "found that the exception I raised for duplicate terms in termdoc fields would cause the writer to crash, so added some exception handling that merely skips the doc being indexed (same as for a too-large term)".
• Test. TestCustomTermFreq.testRepeatedTerms (lucene/core/src/test/org/apache/lucene/index/TestCustomTermFreq.java:106-135) expects the IAE, the cause message "field 'field' has duplicate term 'foo'", then: "ensure IndexWriter remains open and doc was skipped".
• Binary terms are legal on this option. The same {C3} term once per document is indexed fine; IndexOptions.java:52-56 requires each term once per document, not UTF-8 terms.
• A safe helper exists. Term.toString(BytesRef): "If the term is not unicode, the raw bytes will be printed instead" (index/Term.java:104-122). The too-long-term handler in the same try prints raw bytes and never decodes (IndexingChain.java:2076-2094).

The case for intended: repeating a term breaks the option's rule ("Each term can only occur once per document", IndexOptions.java:52-56), so any exception is acceptable. Also raised: the @throws line of Field(String, BytesRef, IndexableFieldType) says it throws if the type "is indexed()" (Field.java:196-197), and IndexOptions is @lucene.experimental (IndexOptions.java:24).

Why it fails: the rule justifies rejecting the document, which Lucene already does without closing the writer for every valid UTF-8 term; nothing documents closing the writer for some byte patterns only. The @throws line is stale: the constructor does not throw for an indexed untokenized type, and the bug reproduces without that constructor through a TokenStream. @lucene.experimental means the API may change, not that misuse may close the writer. Two executed cross-model reviews and a devil's-advocate verifier built this case and all rated the bug real.

Option A was built and checked through gradle on 5adff78bb168; B passed the finder's test only; C was not built.

In TermsHashPerField.add (TermsHashPerField.java:203), build the message with Term.toString(termBytes), which decodes strictly and falls back to the hex bytes of BytesRef.toString() for anything that is not well-formed UTF-8 (Term.java:104-122, BytesRef.java:145-157). Precedent: #7850 (LUCENE-6792) fixed the same mistake in TermsQuery.toString the same way.

-        throw new DuplicateTermException(e.getMessage() + " '" + termBytes.utf8ToString() + "'");
+        throw new DuplicateTermException(e.getMessage() + " '" + Term.toString(termBytes) + "'");

Trade-offs

• Valid UTF-8 terms keep their text ('foo'), so testRepeatedTerms passes unchanged; non-UTF-8 terms print as bytes, '[c3]' and '[41 e2 82]', and the writer stays open (executed in the diagram probe with the fix, assertions on and off).
• Malformed input is reported, not guessed: [80 41] also prints as bytes, '[80 41]', instead of the garbled '\u0080A' (executed).
• Cost only on the error path; no hot-path change.

What the checks validated

• gradle in a scratch worktree at 5adff78bb168, seed DEADBEEF: the repro fails without the fix (assertions on and off) and passes with it (on and off).
• With the fix: the repro (1), TestCustomTermFreq (15), TestTermsHashPerField (3) and TestIndexWriterExceptions (40, 3 skipped): 59 tests, 0 failures, 3 skipped. :lucene:core:checkGoogleJavaFormat passes. The whole core module was not run.
• The adjudication's run: the independent reproducer's test passes with the fix (assertions on and off) and TestCustomTermFreq 15 of 15.

Wrap the decode in try { ... } catch (RuntimeException | AssertionError notUtf8) and print the bytes in the catch (the finder's fix).

Trade-offs

Catches AssertionError, which is unusual; with assertions off a malformed term that does not throw ([80 41]) still yields a garbled message. Passes the finder's test with assertions on; not run through gradle.

Drop the decoration from TermsHashPerField.add and add the term, through Term.toString, where IndexingChain.invertTokenStream turns the exception into the IAE (IndexingChain.java:2095-2097).

Trade-offs

Covers the token-stream path, in two files, but not the BINARY path: IndexingChain.invertTerm has no DuplicateTermException catch, so it would need the same handling, which fix A makes unnecessary. Not built.

Throw the original DuplicateTermException unchanged.

Trade-offs

Loses useful context and changes the cause message that testRepeatedTerms asserts.

Method: semi-formal (a proof-obligation audit of a class's contract, witnessed by an executable repro).

• The finder stated the obligation for TermsHashPerField.add: when positionStreamSlice raises DuplicateTermException, add must raise a DuplicateTermException too, whatever the term's bytes, because IndexingChain keeps the writer open only for that type. Discharging it step by step left one step that can throw: termBytes.utf8ToString(), documented to throw on malformed UTF-8.
• The finder then wrote a public-API JUnit test that fails on main with assertions on and off, and on 10.6.0 and 10.5.0, plus a control probe (ASCII repeat, binary term once, binary term twice).
• An independent reproducer, two executed cross-model reviews, a devil's-advocate verifier and an adjudication reproduced it with their own tests ("Prior reports, history and checks").