ContextSuggestField accepts U+001E and U+001F in contexts, so a context is silently merged into another one or never matches

The ContextSuggestField constructor documents "@throws IllegalArgumentException ... if value or contexts contains any reserved characters". For contexts it checks only CONTEXT_SEPARATOR U+001D, although the same module treats U+001E (HOLE_CHARACTER) and U+001F (SEP_LABEL) as reserved in the suggestion value and in completion query text. The context is indexed as the first token of the completion token stream, so ConcatenateGraphFilter rewrites it: a U+001E byte becomes an epsilon and a U+001F byte is doubled (or removed with preserveSep=false). ContextQuery keys a query context by its raw bytes. So a suggestion indexed under a<1E>b is stored under ab, an exact ContextQuery for ab returns it and labels it ab, and the exact query for a<1E>b returns nothing; a context a<1F>b never matches its own exact query and is reported as a<1F><1F>b. No error, with assertions on and off, in every release since the class appeared in 5.3.0 (LUCENE-6459, 2015). Rejecting the two characters in contexts, like U+001D, fixes it; U+0000 works in contexts and must stay allowed.

• Document suggester: SuggestField / ContextSuggestField index a weighted suggestion with a completion postings format; PrefixCompletionQuery and friends look suggestions up through SuggestIndexSearcher. ContextQuery wraps such a query to filter or boost by context, a string attached to the suggestion (for example a category). addContext("ab") "Adds an exact context"; addAllContexts() matches every context.
• The internal characters: U+001D is ContextSuggestField.CONTEXT_SEPARATOR (ends the context), U+001E is CompletionAnalyzer.HOLE_CHARACTER (= TokenStreamToAutomaton.HOLE, marks a removed token), U+001F is ConcatenateGraphFilter.SEP_LABEL (= TokenStreamToAutomaton.POS_SEP, between tokens). a<1E>b below means the three characters a, U+001E, b.

1. The constructor promises "@throws IllegalArgumentException ... if value or contexts contains any reserved characters" (ContextSuggestField.java:62-63).
2. wrapTokenStream runs validate() on each context, including contexts injected through contexts() (:73-76, :80-83), so the check happens at addDocument. validate() rejects only CONTEXT_SEPARATOR (:166-179).
3. Elsewhere in the module U+001E and U+001F are the reserved characters: SuggestField.isReserved for the value (SuggestField.java:140-149, also U+0000), CompletionQuery.validate for query text ("this character is reserved", CompletionQuery.java:141-154), and the analyzing suggesters (AnalyzingSuggester.java:469, :686, :690).
4. PrefixTokenFilter emits the context plus U+001D as one token ahead of the value (ContextSuggestField.java:138-157), and ConcatenateGraphFilter concatenates the stream. Every token byte becomes an automaton label (TokenStreamToAutomaton.java:197-206), so a literal 0x1E or 0x1F byte cannot be told apart from the HOLE and POS_SEP labels.
5. replaceSep turns every HOLE label into an epsilon (ConcatenateGraphFilter.java:313-323; its comment: "if that's somehow a problem we can always map HOLE to a dedicated byte (and escape it in the input)"), and EscapingTokenStreamToAutomaton doubles every 0x1F byte (:253-281); with preserveSep=false 0x1F labels become epsilons instead (:234-240, :305-312).
6. ContextQuery.addContext also checks only U+001D and keys the query context by its raw bytes (ContextQuery.java:124-144); the exact-match automaton is built from them (:228-250), and the reported context is decoded from the indexed path (:308-331). Index and query side disagree.

A U+0000 context round-trips correctly, so only U+001E and U+001F are affected.

• Who is exposed. Applications that use the document suggester (ContextSuggestField with a completion postings format, queried through ContextQuery) and whose context strings can contain U+001E or U+001F: contexts built from imported data or composite keys that use these characters as delimiters, or taken from document fields through a contexts() override (LUCENE-6702 added that hook to "get context values from another field in a document"). Ordinary category, locale or tenant labels normally do not contain them.
• What goes wrong. Silent wrong results, no error, with assertions on and off: an exact-context filter returns a suggestion indexed under a different context string and labels it with the wrong context, or a suggestion can never be found by its own context. In the tested cases, ordinary contexts, values, weights, and exact, prefix and all-contexts queries without these characters behaved normally, and the index stayed readable and queryable (no CheckIndex run).
• The same cause reaches other context features (a caveat probe on main, assertions on and off): boosts follow the stored bytes (with addContext("ab", 1) and addContext("a<1E>b", 100) the a<1E>b suggestion of weight 9 scores 9, not 900, labelled ab); non-exact (prefix) query contexts match the rewritten bytes (addContext("a<1F>", 1, false) finds the a<1F>b suggestion and reports a<1F><1F>b); a context made only of U+001E becomes the empty context, reported as null and found by an exact query for ""; and the raw contexts ab and a<1E>b on one document collapse into one (all contexts returns one hit, not two).
• Default vs opt-in. The document suggester is an optional module and both classes are @lucene.experimental; within it the trigger needs no setting (preserveSep=false only turns the U+001F case from "unreachable" into "merged").
• Scale and versions. Deterministic, one document is enough. Every release from 5.3.0 to 10.5.2, and main.
• Persistence. A fix stops new occurrences; contexts already indexed in the rewritten form stay that way until the documents are reindexed.
• How likely. Rare: it needs control characters in context strings, and the documented searches found no prior report.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<[s0@ab]> but was:<[s0@ab, s1@ab]>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:120)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.search.suggest.document.TestContextSuggestFieldHoleCharContext.testHoleCharContextIsNotMergedIntoAnotherContext(TestContextSuggestFieldHoleCharContext.java:60)

• The javadoc. The @throws clause names contexts (ContextSuggestField.java:62-63), in the class's first version (be4d58c7740, LUCENE-6459).
• The same commit reserves exactly these characters in the raw suggestion value and in completion query text: SuggestField.isReserved for the value and CompletionQuery.validate ("this character is reserved") for query text; TestSuggestField#testReservedChars asserts that the value rejects [0x1f], [0x1e] and [0x0] (TestSuggestField.java:101-124).
• The analysis comment. replaceSep's comment notes that a removed hole will also match an empty-string token, and that "if that's somehow a problem we can always map HOLE to a dedicated byte (and escape it in the input)" (ConcatenateGraphFilter.java:316-322). Nothing escapes a literal 0x1E in the input, so a context byte 0x1E is removed exactly like a real hole. For the raw value and query text the callers avoid this by rejecting the character; for contexts nobody does.
• Tests and history. The only context test, TestContextSuggestField#testReservedChars (TestContextSuggestField.java:66-103), checks just [0x1d]; no upstream test puts U+001E or U+001F in a context, to allow or to forbid it. No comment on LUCENE-6459 (24 on JIRA, 23 on GitHub) or on the later issues that changed the class discusses reserved characters in contexts, and there is no "won't fix".
• Not a normalisation. If the characters were allowed, the context would at least have to behave as a context; instead the raw string does not find its own document, and a different, already existing context does.

The checkers built the strongest case for "intended": for contexts "reserved" may mean only U+001D. Both validation sites (ContextSuggestField.validate, ContextQuery.addContext) and the only context test check just that character; U+0000, reserved in the value, works in a context, so the reserved set depends on position; so does analyzer output: U+001E or U+001F produced by the application's analyzer in a suggestion value is accepted and round-trips, and contexts bypass the application's analyzer (PrefixTokenFilter adds them after it); the code is unchanged since 2015; both classes are @lucene.experimental; and the input is rare.

Why it fails: nothing documents that a context may silently change, and the API takes any CharSequence. An exact ContextQuery returning a different context breaks the class's stated purpose, "filtering by specified contexts" (ContextQuery.java:42-44). A position-dependent reserved set can explain why U+0000 is harmless in a context, and analyzer output round-trips because the query text goes through the same analysis as the value, so both sides are rewritten alike; a context is rewritten on the index side only, while ContextQuery keys the raw bytes, so it cannot round-trip. Neither explains why U+001E and U+001F, which the same commit reserves in the raw value and in query text, should silently corrupt a context. Rarity and @lucene.experimental are about severity, not intent. Every checker rejected the case; the cross-model reviews rated it weak. A later cross-model fact check also rated it weak overall, but "moderate" for U+001F alone: the analysis escapes U+001F on purpose by doubling it, and an exact query for the doubled form finds the entry; it still fails the raw exact match, changes the reported label, and with preserveSep=false becomes the same merge as U+001E.

Options A and B were applied in a scratch worktree of main a9779ef3c193 and run through gradle with assertions on and off; options C and D were not run.

Extend ContextSuggestField.validate() (ContextSuggestField.java:166-179), which wrapTokenStream already runs on every context, to reject CompletionAnalyzer.HOLE_CHARACTER and ConcatenateGraphFilter.SEP_LABEL as well; +8/-1 lines with the import.

Where it fires. At addDocument, exactly like the U+001D check, so contexts injected through contexts() are covered; the message is the existing "... UTF-16 codepoint [0x1e] at position 1 is a reserved character". The constructor's own validate(value) call is unaffected: SuggestField already rejects both characters in the value.

Timing vs the @throws clause. The clause is on the constructor, but option A, like today's U+001D check, rejects contexts only when the field is analysed (wrapTokenStream, at addDocument), which keeps contexts injected through contexts() covered. A literal reading of the contract would also check the constructor's contexts in the constructor; that is a possible addition (not run).

Not U+0000. It round-trips in contexts, and downstream engines prefix contexts with (char) typeId starting at 0.

+import org.apache.lucene.analysis.miscellaneous.ConcatenateGraphFilter;
 ...
   private void validate(final CharSequence value) {
     for (int i = 0; i < value.length(); i++) {
-      if (CONTEXT_SEPARATOR == value.charAt(i)) {
+      char c = value.charAt(i);
+      // contexts go through the same completion analysis as the value, which drops HOLE_CHARACTER
+      // and escapes SEP_LABEL, so both are reserved in contexts too (the value is already checked
+      // for them by SuggestField)
+      if (c == CONTEXT_SEPARATOR
+          || c == CompletionAnalyzer.HOLE_CHARACTER
+          || c == ConcatenateGraphFilter.SEP_LABEL) {
         throw new IllegalArgumentException(
             "Illegal value ["

Trade-offs

• Applications that index such contexts today get an IllegalArgumentException instead of a silent merge; say so in the release note.
• Contexts already indexed in the rewritten form are not repaired until reindexed.
• No API-surface change (no signature changes), but an intentional behaviour change: input that is accepted today throws. Two extra comparisons per context character.

What the checks validated

• All org.apache.lucene.search.suggest.document.* suites with the new test: stock 62 tests, 4 failed (the new ones); with the fix 62 of 62, assertions on and off.
• All of :lucene:suggest:test with assertions on: 32 classes, 240 tests, 7 skipped, 0 failures.
• :lucene:suggest:check -x test passes (google-java-format, ecjLint, forbiddenApis).
• Through the repro harness: the new test 4 of 4 pass with the fix shadowed on a9779ef3c193. Earlier, on ea57360912ed, the finder's equivalent fix passed the finder's repro and, together with two sibling fix sketches, 136 upstream tests in 11 classes.

The same three-character test in ContextQuery.addContext (ContextQuery.java:128-139), +5/-1 lines. Mirrors CompletionQuery.validate, which already rejects them in query text. Validated together with A: 62 of 62 document tests (assertions on and off), 240 module tests, check -x test passes.

Trade-offs

A query context containing them now throws instead of matching nothing (exact) or the rewritten bytes (prefix), which can surface as an error where applications pass user input as query contexts; it also stops the undocumented exact query for the doubled a<1F><1F>b form that finds an old U+001F context.

Escape U+001F and map U+001E to a dedicated, escaped byte consistently on the index and the query side (for HOLE, this is what the replaceSep comment anticipates).

Trade-offs

Larger: it changes the indexed bytes of these contexts and needs the same escaping in ContextQuery's automaton and in the reported context, for characters that no report found by the searches asks to support.

Say in the javadoc that contexts must not contain U+001E or U+001F.

Trade-offs

The silent merge stays, and the @throws clause would still promise an exception.

• Method (semi-formal): a bounded-exhaustive reject/accept matrix on the stock main jar, in the hunt's unit on completion-suggester indexing, with assertions on and off: every reserved character (U+0000, U+001D, U+001E, U+001F) at every input position (the SuggestField value, the ContextSuggestField value, a context, and the analysis output of either value), 20 cells.
• Oracle. Each cell must either be rejected, as the javadoc documents, or round-trip: the suggestion is found by its own context and by no neighbouring context, and is reported with the context it was given.
• Signal. 18 cells behaved as documented (rejected, or a round trip, including a U+0000 context); the only 2 wrong cells were a context with U+001E (merged into the neighbouring context) and a context with U+001F (unreachable by its exact context). The hypothesis came from modelling the context path of the finder's suggester-level reference.
• Root cause by reading ContextSuggestField.validate(), ConcatenateGraphFilter and ContextQuery.addContext; the finder then wrote the public-API JUnit and a fix sketch, and independent checkers followed (see "Independent checks").