StandardQueryParser keeps escape backslashes in fn: interval term literals

The flexible StandardQueryParser documents that "any character can be quoted with a backslash", and outside interval functions it removes the backslash: a\:b is the term a:b. Inside fn: it does not. The two grammar productions that read an fn: term literal, IntervalText (every literal under fn:or, fn:ordered and the other composite functions, at any depth) and IntervalFuzzyTerm (fn:fuzzyTerm), pass the raw token text to AnalyzedText and FuzzyTerm without discardEscapeChar, which every other production that yields plain term text calls. So fn:or(a\:b) searches for the text a\:b: with a whitespace analyzer it misses the document a:b c and finds a\:b c; with StandardAnalyzer it becomes the phrase BLOCK(a,b) and finds a b c. Nothing throws. The lexer already treats the backslash as an escape inside fn:; the parser just never removes it. Quoting is a workaround except for a double quote, which no fn: term literal can express (only fn:wildcard(a\"b) reaches it, as one unanalyzed term). Every release since 9.1.0, when fn: arrived, is affected; four discardEscapeChar calls fix it.

• Escaping: in StandardQueryParser query text, a backslash makes the next character literal: a\:b is the term a:b, not field a and term b (StandardQueryParser.java:175-186).
• Interval functions: fn:or(...), fn:ordered(...), fn:fuzzyTerm(...) and 18 more build an IntervalQuery; their leaves are term literals, "converted into (analyzed) text intervals" (intervalfn/package-info.java:147-152).

1. The lexer treats \x as an escape everywhere (StandardSyntaxParser.jj:129-136, :159-161); after fn: the ( switches it back to the default state (:147, :167-168), so a\:b inside fn: is one term token with the image a\:b. The escape protects the structure: fn:or(x\)) parses, fn:or(x)) does not.
2. Every other production that yields plain term text removes the backslash with discardEscapeChar: Term (:803-804), QuotedTerm (:828-831), FuzzyOp (:727), the range productions and field names. The only others that keep it, Term's /regexp/ branch (:798-801) and fn:wildcard (:649-650), pass it to RegExp and WildcardQuery, which read it as an escape.
3. IntervalFuzzyTerm (:668-669) and IntervalText (:684-685) do not: they hand token.image (or the quoted image minus its quotes) to FuzzyTerm and AnalyzedText (generated parser :980, :986, :1022, :1046).
4. AnalyzedText passes the text to Intervals.analyzedText and FuzzyTerm to Intervals.fuzzyTerm (AnalyzedText.java:34-41, FuzzyTerm.java:43-50). Neither knows query escapes, so the backslash becomes part of the searched text.
5. IntervalFun (:383-409) is recursive and every term-literal leaf ends in one of the two productions, so every literal under every composite function is affected. fn:wildcard, the one other leaf, keeps backslashes too, but there WildcardQuery reads them as escapes, which is consistent for glob escapes such as \* (its handling of \uXXXX escapes was not tested); it is not part of this bug.

• Who. Applications that pass query text to the flexible StandardQueryParser and use its fn: interval functions, when a term literal inside fn: contains a backslash-escaped character: typed by a user, or produced by the documented helper QueryParserUtil.escape (escape("a:b") is a\:b), for example when an application wraps escaped user words in fn:ordered(...).
• Effect. Silently wrong hits for that literal: it misses the documents with the intended text and can match documents with the backslash text, or, with StandardAnalyzer, the phrase of the split pieces. No exception; the index is not touched.
• Default vs opt-in. Default parser settings, IndexWriterConfig, codec and searcher. The fn: syntax is part of the standard syntax; the trigger is query text that uses it.
• Depends on the analyzer. With whitespace or keyword-style analyzers (codes, identifiers, paths) every escaped character changes the result. With StandardAnalyzer the backslash is usually split off and dropped, so the visible cases are mainly an escaped colon (a:b is one token, a\:b gives a, b), escapes a user types before other in-word punctuation (a\.b is body:a.b, but fn:or(a\.b) is BLOCK(a,b)) and \uXXXX escapes.
• Workaround. Quote without escapes: fn:or("a:b") works. A double quote is the exception: no fn: term literal can express it, since fn:or("a\"b"), fn:or(a\"b), fn:or(a\u0022b) and fn:fuzzyTerm("a\"b" 0) all keep the escape sequence. The only escape hatch is fn:wildcard(a\"b): it finds the a"b document because WildcardQuery reads \" as an escape, but it matches one indexed term without analysis, so it cannot stand in for an analyzed literal or a phrase.
• Scale. None: one document and one query, deterministic, every time.
• Frequency. Not measured, and likely low: it needs the fn: interval syntax plus an escaped character inside an fn: term literal, and quoting usually avoids it. The registry rates the likelihood rare, and no user report was found in the searched sources since 9.1.0.
• Not affected. Escaped terms outside fn:; quoted fn: literals without backslashes; fn:wildcard; the classic QueryParser, which has no fn: syntax.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: fn:or(a\:b) parsed as body:a\:b expected:<1> but was:<0>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:647)
       org.apache.lucene.queryparser.flexible.standard.TestIntervalFunctionEscapedTerm.testEscapedColonInsideFnOr(TestIntervalFunctionEscapedTerm.java:46)

• The contract. StandardQueryParser.java:175-186, "Special character escaping": "If the search term contains the quote character (or cannot be quoted for some reason), any character can be quoted with a backslash", with \:\(quoted\+term\)\: and ":(quoted+term):" as two spellings of one term. The fn: section (:204-223) adds no exception. intervalfn/package-info.java:149: term literals are "converted into (analyzed) text intervals". FuzzyTerm.java:26: "An interval function equivalent to FuzzyQuery", and a\:b~1 outside fn: is unescaped.
• The javadoc example fails inside fn:. fn:or(\:\(quoted\+term\)\:) finds the document with the backslash text; the documented quoted form fn:or(":(quoted+term):") finds the right one (executed by the independent reproducer).
• The grammar disagrees with itself. The lexer honours the escape inside fn:, and Term, QuotedTerm, FuzzyOp, the range productions and field names all call discardEscapeChar. The only other rules that keep the raw image, Term's /regexp/ branch and fn:wildcard, hand it to RegExp and WildcardQuery, which read backslashes as escapes; IntervalText and IntervalFuzzyTerm hand it to the analyzer and the fuzzy automaton, which do not.
• History. Both productions skipped it from their first version (LUCENE-10223, #429, 2021; LUCENE-10414, #668, 2022). The escaping javadoc and the term-literal docs came in the same commit as IntervalText, so code and docs have disagreed from the start. Neither PR discusses escaping.
• Tests. No queryparser test uses a backslash inside fn:; none asserts either behaviour. In #11800 (2022) the fn: author advised escaping (am\@zing) or quoting, with no fn: exception.

The strongest works-as-designed case, built by the intended-behaviour verifier: fn: literals are a raw sub-language. The interval docs never mention escaping, every fn: leaf takes the raw token the same way, fn:wildcard keeps backslashes, and AnalyzedText.toString prints the stored text back unescaped.

Why it fails: the escaping rule is stated for the whole parser and the lexer applies it inside fn:; fn:wildcard and Term's /regexp/ branch keep backslashes only because WildcardQuery and RegExp read them as escapes, while the analyzer and the fuzzy automaton do not; under the raw reading no fn: term literal could contain a double quote (only fn:wildcard, one unanalyzed term, can reach one), exactly the case the javadoc says needs backslash escaping; toString does not round-trip anyway (LFV-748); and no test, commit message, CHANGES or MIGRATE entry, JIRA or PR discussion says fn: literals are raw. All the reviews by a model from another family rated the argument weak. The weakest point: the contract is a general rule, not a sentence about fn:. The double quote supports them.

Option A is the fix for this bug; option B is the PR to send if LFV-748 (toQueryString printing of the same nodes) is filed together. Both were validated in a clean scratch worktree at a9779ef3c193 with gradle (assertions on and off for the repro), and both diffs apply cleanly to that commit.

Side note for the same PR, not counted as a bug. The term-literal example in intervalfn/package-info.java:162 reads fn:or(\"quick fox\") with literal backslashes (Java string escaping leaked into the javadoc). Typed as shown it parses as two unquoted terms with any analyzer: with StandardAnalyzer (the independent reproducer's probe) they are body:or(fox,quick) and match the example sentence, not the phrase that the example says does not match; with a whitespace analyzer they keep the backslash and the quote and match nothing.

Wrap the four token images of IntervalFuzzyTerm and IntervalText in discardEscapeChar(...).toString() (already imported, StandardSyntaxParser.jj:83), then regenerate the parser; the diff also carries the regenerated StandardSyntaxParser.java and its checksum file. fn:wildcard stays as it is on purpose.

// StandardSyntaxParser.jj, IntervalFuzzyTerm (:668-669)
  (<TERM> | <NUMBER>) { term = discardEscapeChar(token.image).toString(); }
| <QUOTED>            { term = discardEscapeChar(token.image.substring(1, token.image.length() - 1)).toString(); }
// IntervalText (:684-685)
  (<QUOTED>)          { return new AnalyzedText(discardEscapeChar(token.image.substring(1, token.image.length() - 1)).toString()); }
| (<TERM> | <NUMBER>) { return new AnalyzedText(discardEscapeChar(token.image).toString()); }
// then: ./gradlew :lucene:queryparser:regenerateFlexibleStandardSyntaxParser

Trade-offs

Escaped fn: literals now mean the unescaped text; anyone who relied on the raw text (no such use is documented) sees different hits, worth a CHANGES line. Malformed unicode escapes inside fn: (fn:or(a\u00)) now throw ParseException, as they already do outside fn:. Alone, it makes toQueryString print escaped literals in a form that no longer parses back (fn:or(a\:b) prints fn:or(a:b), a ParseException; fn:or(x\\y) prints fn:or(x\y), which re-parses as xy): pair it with option B's printing.

What the checks validated

Stock: the two repro files fail 9 of 10 (assertions on and off), and the whole module run fails exactly those 9. Fix: 10 of 10 pass (on and off); the whole lucene/queryparser suite (544 tests: 533 upstream + 10 repro + 1 probe; 0 failures, 1 upstream @Ignore) and the highlighter's TestMatchHighlighter (6) pass; :lucene:queryparser:check -x test (javacc checksums, google-java-format, ecjLint, forbiddenApis) passes. The regenerated parser equals the hunt's hand-patched one up to line wrapping; with that one shadowed onto the stock jars all 11 scope-probe queries hit doc 0 and the hunt's exhaustive escape check goes from 0 to 5,208 of 5,208 equal.

Option A plus the LFV-748 printing fix (quote every literal the grammar would not read back as one term; fn:fuzzyTerm prints %d %d; fn:wildcard prints a bare maxExpansions) and one helper, AnalyzedText.toLiteral, that escapes \ and " inside the quotes; FuzzyTerm.toString uses it. With it every parsed fn: query in the probe prints a string that parses back to the same query.

static String toLiteral(String term) {  // AnalyzedText; FuzzyTerm.toString uses it too
  if (requiresQuotes(term)) {            // requiresQuotes as broadened by the LFV-748 fix
    return '"' + term.replace("\\", "\\\\").replace("\"", "\\\"") + '"';
  } else {
    return term;
  }
}

Trade-offs

toString/toQueryString now quote literals with query-syntax characters. fn:wildcard keeps its raw glob inside the quotes, because the wildcard rule does not unescape. Filed separately, A is this bug's fix and the printing belongs to LFV-748; say in each that they land together, and do not count the printing failures twice. Neither diff adds a test: the round-trip probe prints whether each re-parse equals the original and asserts nothing, so a PR should add the repro tests and a round-trip assertion.

What the checks validated

Same worktree: 9 of 9 round-trip probe inputs print a string that re-parses to the same query (fn:or(a\:b) -> fn:or("a:b"), fn:or(x\\y) -> fn:or("x\\y"), fn:fuzzyTerm(a\:b 0) -> fn:fuzzyTerm("a:b" 0 128)); the whole lucene/queryparser suite (544, 0 failures, 1 skipped) and :lucene:queryparser:check -x test pass; tidy only rewrapped one boolean chain.

One place instead of four and no parser regeneration.

Trade-offs

These are public node classes that programmatic callers build from plain text; unescaping there would change what new AnalyzedText("a\\b") means for them. Not run.

Keep the code and document the exception.

Trade-offs

Contradicts the parser's escaping section, leaves its own example wrong inside fn:, and leaves no fn: term literal that can hold a double quote. Not run.

Method: bounded-exhaustive translation validation (semi-formal tier), in a hunt unit on the standard query parser's documented syntax.

• The reference prints each fn: expression from an abstract syntax tree in the documented syntax and builds the IntervalsSource with the Intervals factory of the same name, unescaping term literals by the documented rule; it compares the parser's result by Query.equals and by match sets and positions on a 389-document two-field index.
• Plain scope: 11,520 exhaustive checks (12 leaves, every depth-1 function, 3 field modes) and 75,000 random depth-2 and depth-3 checks, 0 mismatches; four deliberately broken references were all caught.
• Escape dimension (7 escaped literals): 5,208 checks, 0 equal, 2,250 with different match sets; with the four-call fix 5,208 of 5,208 equal.
• The signal came from reading the grammar while writing the reference (IntervalText has no discardEscapeChar); the run measured it. Found 2026-10-27, then checked by an independent reproducer, reviews by a model from another family (OpenAI GPT-5.6, reading only), an intended-behaviour devil's advocate, an adjudication, a history and prior-report follow-up and a credibility audit.