Surround query parser keeps the escaping backslashes inside a quoted term, so "a\"b" searches for a\"b instead of a"b

In the surround query language a term in double quotes is one literal term. Inside the quotes the lexer accepts a backslash only as \" or \\ and rejects a lone backslash, and the grammar comment on the token says "escape quoted quote or backslash by backslash". But the parser action for the token only removes the outer quotes, so SrndTermQuery searches the text with the escaping backslashes still in it: "a\"b" searches a\"b and "c:\\temp dir" searches c:\\temp dir. The intended term gets 0 hits, or a term that literally contains the backslashes matches instead; nothing throws. Since a quote cannot appear in an unquoted term and a lone backslash cannot appear inside quotes, no surround syntax matches such terms exactly. The classic and flexible parsers use a similar, broader quoted-token idiom and do remove the escapes. The token and its action date from the 2005 contribution and are semantically unchanged; every release checked from 1.9 to 10.5.2, 10.6.0 (by reading), and main, is affected. Unescaping in the parser and re-escaping in SrndTermQuery.toString fixes it and keeps parse(toString()) round-tripping.

1. Lexer (QueryParser.jj:223-228): the comment "Literal non empty term between single quotes, escape quoted quote or backslash by backslash." sits above the token <QUOTED: "\"" ( (~["\"", "\\"]) | ("\\" ["\\", "\""]))+ "\"">. Inside the quotes a character is anything except " and \, or a backslash followed by \ or ". A lone backslash, as in "p\q", is a ParseException.
2. Parser action (QueryParser.jj:429-430, generated QueryParser.java:479-487): getTermQuery(term.image.substring(1, term.image.length()-1), true /* quoted */). Only the outer quotes are cut off; nothing removes the backslash of \" or \\.
3. Query (SrndTermQuery.java:40-41, :50-61): getTermQuery builds new SrndTermQuery(text, true), which seeks exactly that text (seekCeil, :57). When the text is not in the index the query rewrites to MatchNoDocsQuery (SimpleTermRewriteQuery.java:48).

Five documents with one untokenized StringField value each, searched with QueryParser.parse(text).makeLuceneQueryField("k", new BasicQueryFactory()):

Without the raw-escaped documents both queries return 0 hits instead of 1. On a WhitespaceAnalyzer field, which keeps the token "hello" with its quotes, said w "\"hello\"" finds 0 of 1 documents.

• An unquoted term cannot contain " (_TERM_CHAR, QueryParser.jj:194-196), so a"b is a lexical error.
• Unquoted, c:\temp dir is two terms and c: is the field operator: c:\temp parses as field c, term \temp.
• The truncated form "c:\temp dir"* (TRUNCQUOTED, :227, no escapes) keeps backslashes as typed, but it is a prefix query and also matches longer terms. It cannot contain a quote ("a\"b"* is a lexical error), so for a term with a quote only a prefix that stops before the quote is left: "say "* matches say "hi", and every other term that starts with say .
• ? can stand in for an inner quote (abc?def), but it also matches any other character, and it needs at least 3 other characters: a?b is rejected ("Too unrestrictive truncation").
• An unquoted term with a backslash already works: a\b matches a\b.

• Who. Callers of QueryParser.parse / parse2 and makeLuceneQueryField in org.apache.lucene.queryparser.surround: an opt-in, niche parser; open issue #16820 proposes deprecating it for Lucene 11. Solr's SurroundQParserPlugin passes the query string straight to QueryParser.parse and makeLuceneQueryField (read on apache/solr main, not executed).
• Which terms. An exact value that contains a double quote (12" pipe, say "hi", or a whitespace token that keeps its quotes), or a value with a backslash that also needs quotes because of a space, : or an operator character (c:\temp dir). Users are pointed to exactly this syntax: the one answer to Stack Overflow question 16935363 (2013, Solr 4.0) recommends "sec\"ond".
• Default vs opt-in. No setting is involved: a probe with a plain new IndexWriterConfig(), the default codec, a plain IndexSearcher and BasicQueryFactory (its 1024-clause limit still applies) gives the same wrong match; the compact repro runs under the test framework's randomized configuration. Only the parser is opt-in.
• Always, silently. The term text is wrong before the index is touched, so it happens every time, with assertions on and off, at any scale: 0 hits, or hits on a different term that literally contains the backslashes. No exception.
• Likelihood: rare. Such terms are unusual in fields searched with surround, and the only public trace the searches found in 21 years is the 2013 question.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: hits for the term a"b expected:<1> but was:<0>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:647)
       org.apache.lucene.queryparser.surround.query.TestSrndQuotedTermEscapes.testEscapedQuoteInQuotedTerm(TestSrndQuotedTermEscapes.java:44)

• Grammar. QueryParser.jj:223-225: "Literal non empty term between single quotes, escape quoted quote or backslash by backslash. Evt. truncated." (it says single quotes, but the token uses double quotes). The token admits a backslash only as such an escape. With raw semantics "p\q" would be unambiguous, yet it is rejected, and requiring \\ would make every quoted single-backslash term impossible.
• Docs and intent. The class javadoc lists "quoting (with ")" (QueryParser.java:27). The parser's author wrote on java-user on 2004-06-09 ("Syntax for query parsers"): "double quotes can be used to search for any single term". A term with a quote that is not preceded by a backslash, such as a"b, cannot be searched exactly today.
• Sibling parsers. Classic QueryParser.jj:188, 207 define QUOTED with a similar, broader idiom (a backslash may escape any character, :182, and the body may be empty), and QueryParserBase.handleQuotedTerm calls discardEscapeChar on the inner text (QueryParserBase.java:809), documented as "Returns a String where the escape char has been removed, or kept only once if there was a double escape" (:834-840). The flexible StandardSyntaxParser.jj:831 does the same. Surround has the lexical half, restricted to \" and \\, without the decoding.
• Precedent. LUCENE-573 ("Escaped quotes inside a phrase cause a ParseException", 2006) and LUCENE-1189 ("QueryParser does not correctly handle escaped characters within quoted strings", 2008) were fixed as bugs in the classic parser.
• Tests and history. Test01Exceptions, Test02Boolean, Test03Distance and TestSrndQuery never put an escape inside quotes; with the fix all 67 pass. The token and its action are semantically unchanged since the 2005 contribution. No CHANGES or MIGRATE entry says the backslashes are meant to stay, and the searches found no issue or list thread that does.

The cross-model reviews (refute and intended lens) built the strongest case for "intended" and rated it moderate; the devil's advocate built the same case and concluded that none of it survives. The case: surround can be read as raw, the text between the quotes being the term, with the backslash rule only keeping the token from ending early. Unquoted terms and the truncated form "..."* also keep backslashes as typed. Keeping the raw text makes parse(q).toString() round-trip exactly, which matters because surround query equality is toString() equality (LUCENE-2945). No complaint was found in 21 years, a fix changes what existing queries with \\ mean, and the protected getTermQuery(String, boolean) hook lets an application decode itself.

Why it fails: under the raw reading ordinary terms cannot be written exactly (a term with a quote that is not preceded by a backslash, such as a"b; a quoted term with one backslash, such as c:\temp dir), the lexer's rejection of a lone backslash would serve no evident purpose, and the grammar's "escape" comment would describe nothing. That is strong evidence, not proof. The round trip is kept by re-escaping in toString (the validated fix). Age without reports reflects how niche the parser is; the hook is a workaround, and the javadoc says parse() is the only method clients should need. Verdict: a defect. Upstream may still prefer to document raw semantics, but the impossible terms would remain.

One fix was built and validated in a clean worktree; the parser-only variant from the hunt is shown for comparison. Any decoding fix changes what existing quoted queries with \\ or \" mean, so it needs a CHANGES entry.

In QueryParser.jj, a static unescapeQuoted(String) drops the backslash of each \" and \\ (the lexer guarantees that a backslash inside QUOTED is followed by one of the two), and the QUOTED action calls it; QueryParser.java and its checksum file are regenerated with ./gradlew :lucene:queryparser:regenerateSurroundQueryParser. In SrndTermQuery.toStringUnquoted(), a quoted term writes \ as \\ and " as \", so parse(q).toString() is unchanged and still parses back to an equal query.

   | term=<QUOTED>
-    {return getTermQuery(term.image.substring(1, term.image.length()-1), true /* quoted */);}
+    {return getTermQuery(unescapeQuoted(term.image.substring(1, term.image.length()-1)), true /* quoted */);}

   // SrndTermQuery.toStringUnquoted()
+    if (isQuoted()) {
+      return getTermText().replace("\\", "\\\\").replace("\"", "\\\"");
+    }
     return getTermText();

Trade-offs

• Meaning change. "a\\b" then searches a\b instead of a\\b, and "a\"b" searches a"b. Worth a CHANGES line.
• Hook input. Subclasses that override the protected getTermQuery(String, boolean) now receive the decoded text for quoted terms.
• Direct construction. A SrndTermQuery built by hand with quoted=true and a backslash or quote in its text now prints with escapes, so its toString, equals and hashCode change: it equals a parsed query only when its text is the decoded form. What it matches does not change.
• Truncated form left as is. TRUNCQUOTED ("..."*) has no escapes and keeps a lone backslash, so "a\\b" would mean a\b while "a\\b"* still means the prefix a\\b. Upstream can decide whether to give it the same escapes; that would turn today's "x\y z"* into a lexical error.
• Tests to add upstream. By inspection, unescapeQuoted and the re-escaping are inverses on every body the QUOTED token accepts, but the round-trip guard runs only three strings: add adjacent and repeated escape pairs and directly constructed queries.

What the checks validated

• Stock main a9779ef3c193, gradle in a clean worktree: the repro fails 2 of 4 with assertions on and with assertions off.
• With the fix: the whole lucene/queryparser suite passes (31 classes, 537 tests, 0 failures, 1 skipped by an existing @Ignore), including the repro 4 of 4 and the four surround suites (67 tests); the repro also passes 4 of 4 with assertions off.
• With the fix: checkGoogleJavaFormat, ecjLintMain, ecjLintTest, forbiddenApisMain, forbiddenApisTest and regenerateSurroundQueryParserChecksumCheck pass; QueryParser.java was regenerated from the grammar by the project's own task.
• Fix shadowed onto the stock jars: the finder's test 3 of 3 and the independent reproducer's 7 of 7 pass; the devil's advocate's 4 bug tests and its round-trip check pass, and only its info test that records today's behaviour flips, as expected.
• Not run: a benchmark (the helper returns its input unchanged when it has no backslash).

One replaceAll on the text between the quotes. Fixes matching: the finder's, the reproducer's, both reviewers' and the devil's advocate's bug tests pass, and the 67 surround tests pass.

Trade-offs

Breaks the toString round trip, so query equality changes: the parsed query "a\"b" then prints as "a"b", which does not parse back (this repro's guard fails: expected:<"a[\]"b"> but was:<"a[]"b">). Not recommended alone.

Change the grammar comment and the javadoc to say that backslashes inside quotes are kept.

Trade-offs

No compatibility change, but a term with a quote that is not preceded by a backslash (a"b), or a quoted term with a single backslash (c:\temp dir), stays impossible to search exactly. Not explored: also accepting a lone backslash inside quotes (fixes "c:\temp dir", not a"b), or changing nothing and leaving it to the deprecation discussed in #16820.

• Method: non-formal code reading. The hunt unit checked surround query parsing against a reference semantics with a bounded-exhaustive translation-validation harness. Its check passed for quoted terms without escapes, so the harness did not give the signal.
• While writing the harness's renderer for quoted leaves, the finder read the lexer's QUOTED token, saw that it admits \" and \\, and found no code that removes them.
• A probe on the stock main jar showed "a\"b", "a\\b" and "c:\\temp" rewriting to MatchNoDocsQuery (0 hits), while unquoted a\b matched and the classic parser decoded the same quoted escapes. A public-API JUnit followed (2026-10-27).
• An independent reproducer, reviewers from another model family (OpenAI GPT-5.6) and an intended-behaviour devil's advocate then reproduced it with their own tests, and an adjudication confirmed it with new seeds.