The document suggester checks its reserved end byte U+0000 only in the raw value, so an analyzer that emits U+0000 hides suggestions

The document (NRT) suggester stores every suggestion in an FST whose input is the analyzed form, then the end byte 0x00 (NRTSuggesterBuilder.END_BYTE), then one or more dedup bytes. Because 0x00 ends every entry, SuggestField reserves U+0000 and throws IllegalArgumentException for a value that contains it. It checks only the raw value, though; the analyzed form, which is what goes into the FST, is never checked, and the class javadoc invites "the usual Analyzers".

A stock char filter turns an accepted value into a reserved one: HTMLStripCharFilter decodes � to U+0000, and a MappingCharFilter rule can do the same. Then the input of a (61 00 00) is a prefix of the input of a+U+0000 (61 00 00 00), and a second entry for a (61 00 01) sorts after a+U+0000+U+0000 (61 00 00 00 00). In production (assertions off) suggestions silently disappear: with a (weight 1) and a� (weight 2) in one segment, a prefix query for a with n = 100 returns only the second document, and with a��, a, a, b a prefix query for a�� misses the first document (a query for a still lists it). With assertions on, the second case fails the commit with FSTCompiler's "inputs are added out of order" AssertionError, which closes the IndexWriter. The same analyzer with  (U+0001) indexes and finds everything. The bug came with the document suggester itself (5.1.0, 2015) and is in every release since; checking the analyzed form in CompletionTokenStream and throwing the same non-aborting IllegalArgumentException fixes these symptoms. The same in-band byte also gives a wrong extra hit on the query side, which that check does not remove: on main, a prefix query for a� also returns a.

• Document suggester: SuggestField (or ContextSuggestField) indexed with a completion postings format such as Completion104PostingsFormat, queried with SuggestIndexSearcher and a CompletionQuery such as PrefixCompletionQuery. It is @lucene.experimental and opt-in.
• Analyzed form: the suggestion after the field's analyzer, concatenated by ConcatenateGraphFilter inside CompletionTokenStream. The FST is built from these bytes (UTF-8; U+0000 is the byte 0x00).
• FST input: analyzed bytes + END_BYTE 0x00 ("Marks end of the analyzed input and start of dedup byte", NRTSuggesterBuilder.java:42-43) + a dedup byte 0, 1, 2, ... per entry with that analyzed form (:66-69, :81-97). FST inputs must be added in increasing byte order.

1. Only the raw value is checked. The constructor loops over the value (SuggestField.java:86-96) and rejects isReserved characters: U+001F, U+001E and NRTSuggesterBuilder.END_BYTE U+0000 (:140-149), as its javadoc promises (:75-76).
2. The analyzed form passes unchecked. SuggestField.tokenStream wraps the analyzer's stream in CompletionTokenStream (:103-120), whose incrementToken passes the ConcatenateGraphFilter output on unchanged (CompletionTokenStream.java:74-81). HTMLStripCharFilter has already decoded � (or �) to U+0000: any code point up to 0x10FFFF except surrogates goes through Character.toChars (HTMLStripCharFilter.jflex:494-514, :528-547).
3. 0x00 is the terminator, unescaped. At flush CompletionFieldsConsumer calls startTerm once per analyzed term, in byte order, then one entry per document and finishTerm (CompletionFieldsConsumer.java:195-227). So a with one entry is 61 00 00, with two entries 61 00 00 and 61 00 01; a+U+0000 is 61 00 00 00; a+U+0000+U+0000 is 61 00 00 00 00. Without a 0x00 inside an analyzed form the inputs are prefix-free and increasing; with it, both properties break.
4. Out of order at flush. a's second input 61 00 01 comes before 61 00 00 00 00 although it is larger. FSTCompiler.add asserts increasing order (FSTCompiler.java:854-855). With assertions on, the AssertionError escapes flush (NRTSuggesterBuilder.finishTerm:96 <- CompletionFieldsConsumer:226 <- IndexWriter.commit), a tragic event that closes the writer. With assertions off the FST is compiled from out-of-order inputs and a query for a�� no longer returns that document.
5. A hidden suggestion at lookup. 61 00 00 is now both the end of a and a step on the way to a+U+0000. NRTSuggester collects only in its acceptResult override of Util.TopNSearcher (NRTSuggester.java:189-227; "we do not use it, instead collect at acceptResult", :254-255). When the higher-weight a+U+0000 wins at that node (weights are stored as Integer.MAX_VALUE - weight, lower is better, :329-334), a's final path is queued; when it is popped, TopNSearcher adds it to its own result list without calling acceptResult (Util.java:333-337). That branch is a separate weakness, reachable here only because the inputs stopped being prefix-free.
6. When it triggers. Some analyzed form in a segment must start with another analyzed form of the same segment followed by U+0000, such as a and a+U+0000 (from the encoding). A lone x� with no plain x in the segment is indexed and found normally. Merges rebuild the FST from the merged terms, so a pair in two segments meets again at merge. The extra hit below needs only a query whose analyzed form ends in U+0000 and a suggestion equal to the text before it.

Analyzer HTMLStripCharFilter + KeywordTokenizer, one document per value with weight = position + 1, one segment, PrefixCompletionQuery with n = 100 for each value (gradle on main, seed DEADBEEF):

The hidden case needs the plain suggestion to have the lower weight: weights (1,2) and (1,10) lose a; (2,1), (5,5) and (10,1) do not (independent reproducer).

In the last case, with assertions off, a prefix query for a still lists doc 0; only the query for its own value misses it (diagram probe). Extra hits happen too, on the query side: with a (weight 2) and a� (weight 1), a prefix query for a� (analyzed 61 00) also returns a, which does not start with a+U+0000, because a's FST input 61 00 00 starts with the query bytes. The same happens with the fix in place, where only a is indexed (diagram probe, assertions on and off). It is the same class as the registered LFV-691 (RegexCompletionQuery's . matches the end byte), reached here through the query's own analyzed U+0000.

A MappingCharFilter rule ~ -> U+0000 hides a in the two-document case with assertions on and off, and fails the commit of a, a, a~ with assertions on; with them off that three-document case commits and finds all three (independent reproducer).

• Who is hit. Users of the document suggester (SuggestField or ContextSuggestField with a completion postings format; opt-in, @lucene.experimental) whose analysis chain can turn an accepted value into a form with U+0000. Stock components and plain document text suffice: HTMLStripCharFilter decodes � and �; a MappingCharFilter or PatternReplaceCharFilter rule can output U+0000. Common text analyzers such as StandardAnalyzer add no U+0000. Binary-term analyzers such as CollationKeyAnalyzer write raw collation-key bytes, which contain 0x00 by design (a JDK root-locale collation key for a is 00 53 00 00 00 01 00 00 00 01); how they behave in a suggest field today was not tested.
• Silent in production. With assertions off nothing throws: a suggestion accepted at addDocument is just missing from the results of a query for its own value. The lost one can be the U+0000 document or an ordinary neighbour with a lower weight. A query whose analyzed form ends in U+0000 can also return suggestions that do not start with that form.
• Loud with assertions. With -ea (tests, development) the commit throws and closes the IndexWriter; every document added since the last commit is lost. A merge that brings such a pair together trips the same assertion, and IndexWriter.merge handles any merge failure as a tragic event that closes the writer (IndexWriter.java:5109-5113); in the probe's four such merges, forceMerge three times reported the writer's unrecoverable error and once reported the merge exception while isOpen() still returned true. Segments committed before are not lost.
• Scale. Deterministic at any index size, given the colliding pair in one segment. In the hunt's bounded-exhaustive run over every list of 1-3 documents whose analyzed forms are 1-3 characters over {a, U+0000} starting with a (399 lists): with assertions on, 12 flush failures (each closes the writer) and 68 lists with a missed suggestion; with them off, 0 and 71; 52 of the missed queries are for documents without U+0000; the U+0001 control has 0 and 0. An independent re-implementation in the fact check got the same counts and found the 12 failing flushes to be exactly the 12 lists predicted out of order. In a larger probe (2 to 600 copies of a with weights 1..n, plus U+0000 forms with the highest weights, single and merged segments, assertions off) 76 of 78 runs lost U+0000 documents and none lost an a document. These are counts over the probe's constructed cases, not a frequency estimate; in other layouts a plain document is the one hidden (the single a above, the 52 missed queries). No run threw, ran out of memory or took longer than the probe's 20 s limit per operation; allocation was not measured.
• Likelihood. Rare: text with � (or a mapping rule to U+0000) next to a suggestion equal to the text before it, plus a weight order or duplicate count that exposes it. No prior report was found in the recorded searches (2026-10-27; see Prior reports).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<[1, 0]> but was:<[1]>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:120)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.search.suggest.document.TestSuggestFieldAnalyzedEndByte.testAnalyzedEndByteHidesOtherSuggestion(TestSuggestFieldAnalyzedEndByte.java:53)

• The reservation exists for the FST. U+0000 is in SuggestField.isReserved (SuggestField.java:140-149) because it is NRTSuggesterBuilder.END_BYTE, which "Marks end of the analyzed input" (NRTSuggesterBuilder.java:42-43). Only the analyzed bytes reach the FST, so a check on the raw value alone cannot keep the byte out.
• Any analyzer is invited. "Besides the usual Analyzers, CompletionAnalyzer can be used" (SuggestField.java:35-37). No javadoc, CHANGES or MIGRATE entry restricts analyzer output.
• In-band markers are escaped elsewhere. ConcatenateGraphFilter escapes its own SEP_LABEL inside tokens; nothing handles END_BYTE.
• Tests cover the raw value only. TestSuggestField#testReservedChars (TestSuggestField.java:101-124) builds SuggestFields with raw reserved characters. No document-suggester test (suggest/document) uses an analyzer that emits U+0000; the only 0-byte analyzer test in lucene/suggest is TestAnalyzingSuggester#test0ByteKeys (TestAnalyzingSuggester.java:1204-1223), added by LUCENE-4534 (#5600) for the AnalyzingSuggester.
• Precedent. #5600 / LUCENE-4534 (2012) fixed 0 bytes in AnalyzingSuggester/WFST keys as a bug ("0 bytes are rare, they are 'allowed' but will cause exceptions"), with test0ByteKeys as its regression test.
• History. LUCENE-6339 (#7400) introduced the encoding; LUCENE-6459 (#7518) added the raw-value reservation of U+0000. Neither review discusses analyzer output.
• The control. The same analyzer and documents with  (U+0001) index and find every suggestion.

The strongest case for "intended": the API is @lucene.experimental; the javadoc promises the IllegalArgumentException for the raw value only; the writer close needs assertions; HTML5 maps � to U+FFFD, so one could blame HTMLStripCharFilter; and NRTSuggester says "search admissibility is not guaranteed" (NRTSuggester.java:257).

Why it fails: experimental allows API changes, not a silent loss of other documents' suggestions; the production (assertions off) loss is the main symptom, not the writer close; a MappingCharFilter rule triggers the same defect, so the missing check is the bug whatever HTMLStripCharFilter should decode � to; and the admissibility notes tie misses to many deletions or a restrictive filter (:55-64, :280-290), while the repro has neither and asks for 100 hits out of 2-4 documents. Binary-term analyzers (collation keys) emit 0x00 by design; whether a suggest field should accept them is a decision for the fix, not an argument that the silent loss is intended. Two reading-only reviews by one model from another family (OpenAI GPT-5.6; neither ran code) rated the intended argument weak; a later fact check by the same family rated the strongest case moderate for the U+0000 suggestion's own miss and for the assertions-on writer close, and weak for the hidden neighbour. The devil's advocate found no intended or misuse argument that survives. Verdict: a defect.

The recommended fix was applied in a scratch worktree at a9779ef3c193 and run through gradle (seed DEADBEEF), and earlier shadowed onto the stock jars in the hunt. The other options were not run, except that the hunt shadowed a TopNSearcher change alone.

In CompletionTokenStream.incrementToken, scan the term bytes of each concatenated form and throw IllegalArgumentException if one is NRTSuggesterBuilder.END_BYTE, as SuggestField does for the raw value. The exception is thrown during addDocument, so it is non-aborting: the document is rejected and the writer stays open. It covers ContextSuggestField too, whose contexts are part of the same bytes. It checks the index side only (see the trade-offs).

BytesRef analyzed = termAttr.getBytesRef(); // TermToBytesRefAttribute
for (int i = 0; i < analyzed.length; i++) {
  if (analyzed.bytes[analyzed.offset + i] == NRTSuggesterBuilder.END_BYTE) {
    throw new IllegalArgumentException(
        "analyzed form contains the reserved byte 0x"
            + Integer.toHexString(NRTSuggesterBuilder.END_BYTE)
            + " at position "
            + i);
  }
}
payloadAttr.setPayload(payload);

Trade-offs

• Behaviour change. It rejects every analyzed form with a 0x00, also a lone x� that works today, and any analyzer that emits 0x00 on purpose (for example binary or collation-key terms) can no longer feed a suggest field. It also rejects a raw context that contains U+0000, which ContextSuggestField accepts today (its own check, ContextSuggestField.java:166-179, rejects only U+001D; not tested). Say so in a CHANGES entry; the javadoc of SuggestField could mention that the analyzed form is checked too.
• Cost. One pass over the bytes of each suggestion at index time; nothing at search time. Not benchmarked.
• Existing indexes. The check runs only when a document is added. Merges copy the stored terms through CompletionFieldsConsumer (CompletionFieldsConsumer.java:85-102, :195-226) without the token stream, so a segment written before the fix keeps its entries: suggestions stay missing, and with -ea a later merge can still trip the assertion. Only reindexing the affected documents removes them.
• Query side not covered. A prefix query whose analyzed form ends in U+0000 still matches suggestions whose analyzed form is the query's without that last U+0000: with the fix, an index holding only a returns a for a query for a� (diagram probe, assertions on and off). Rejecting or escaping 0x00 in query analysis as well would close that; not run. The repro does not test it. LFV-691 needs a query-side fix for RegexCompletionQuery too; a shared change to the query automaton may cover both (not explored).
• Message. It could also name the field, and the value as the raw-value message does.
• Add a regression test: fix-with-test.diff adds TestSuggestField#testReservedCharInAnalyzedForm.

What the checks validated

• Scratch worktree at a9779ef3c193, gradle, seed DEADBEEF, 2026-10-28. Stock: the repro fails 2 of 3 with -Ptests.asserts=true and with =false; the new regression test TestSuggestField#testReservedCharInAnalyzedForm fails ("Expected exception IllegalArgumentException but no exception was thrown").
• With the fix: the repro passes 3 of 3 and TestSuggestField 21 of 21 (with the new test), assertions on and off; the whole lucene/suggest module with the repro added, assertions on: 32 classes, 240 tests, 7 skipped, 0 failures (the suggest.document package: 62 tests); checkGoogleJavaFormat, ecjLintMain, ecjLintTest, forbiddenApisMain and forbiddenApisTest pass.
• The stock gradle runs used the repro before an unused catch parameter was renamed to _ for the lint; the final file fails 2 of 3 on stock main (assertions on and off) and on 10.5.2 with a standalone runner, and passes 3 of 3 with the fix shadowed (seed 5EED0798).
• Earlier, in the hunt (fix classes shadowed onto the stock jars): the finder's JUnit 3 of 3 and the independent reproducer's 5 of 5 pass with assertions on and off; the 58 upstream suggest/document tests pass; the bounded-exhaustive harness shows 0 flush failures and 0 misses.
• Not run: a benchmark; other modules.

Like ConcatenateGraphFilter escapes SEP_LABEL: index an in-band 0x00 as an escape sequence and build every completion query's automaton the same way.

Trade-offs

Keeps such values indexable and findable and also covers the query-side extra hit, but touches the FST content and every CompletionQuery (prefix, regex, fuzzy, context), so it is a larger change with format implications. Not run.

HTML5 maps the character reference for code point 0 to U+FFFD.

Trade-offs

Removes one trigger only: a MappingCharFilter or PatternReplaceCharFilter rule still produces U+0000. It also changes the filter's output for every user. A possible separate hardening, not a fix of this bug. Not run.

TopNSearcher adds a queued END_LABEL path to its results without acceptResult (Util.java:333-337), which is how the lower-weight suggestion is lost.

Trade-offs

Partial: the hunt shadowed such a change alone, and the hidden-suggestion test passed but the out-of-order flush still failed (3 run, 1 failed). Worth its own issue; it does not stop 0x00 from reaching the FST.

A javadoc note on SuggestField and CompletionAnalyzer.

Trade-offs

Leaves the silent loss and the writer close in place for anyone who misses the note. Not recommended.

Method: bounded-exhaustive differential checking on the stock main jar (the hunt's "semi-formal" tier; no model checker, solver or prover).

• An earlier hunt over the completion suggester had seen the mechanism with a custom MappingCharFilter rule, found no stock trigger and did not report it. A later unit that turns such stranded leads into public-API repros picked it up.
• The harness indexed every list of 1-3 documents whose suggestion analyzes to a 1-3 character string over {a, U+0000} starting with a (399 lists), with U+0001 as the control, and compared the jar with two oracles: an independent reference (every accepted document is found by a prefix query for its own value; commit succeeds) and an independent model of NRTSuggesterBuilder's FST input order, which predicts exactly the 12 lists whose flush fails with assertions on (a count match in the hunt; the fact check's independent re-implementation confirmed the same set).
• The finder then searched for a stock trigger, found HTMLStripCharFilter decoding �, and wrote the JUnit (2026-10-27). An independent reproducer, reading-only reviews by a model from another family (OpenAI GPT-5.6), an intended-behaviour devil's advocate, an adjudication, a history sweep, a prior-report search, a security screen and a credibility audit followed.