Flexible query parser: QueryNode#toQueryString prints field names without escaping, so 'my\ field:x' comes back as 'my field:x', a query on other fields

The standard flexible query parser accepts escaped field names on purpose: my\ field:x is a term query on the field my field and a\:b:x one on a:b. QueryNode#toQueryString is documented to return "a query string understood by the query parser", and the node package says a tree "can also generate a query string that can be parsed back by the original text parser". But every toQueryString that prints a field appends the name raw; the field never goes through the EscapeQuerySyntax. So the node of my\ field:x prints my field:x, which parses as df:my OR field:x, and the node of a\:b:x prints a:b:x, a syntax error; phrase, fuzzy, hand-built nodes and, on main, range nodes do the same. Of 9,119 printable-ASCII field names of length 1 or 2 (all but *), 3,384 fail. It has been there since the flexible parser's first release, 2.9.0; StandardQueryParser.parse builds the right Query, and only code that prints node trees back to text is affected. A validated fix escapes the field through one helper at the eight field-printing sites of the nodes the standard parser builds; two more raw prints, in nodes no parser builds and whose output does not parse back anyway, are left as they are.

FieldQueryNode.toQueryString (FieldQueryNode.java:65-71) returns this.field + ":" + getTermEscaped(escaper) (:69); getTermEscaped (:56-58) runs only the text through escaper.escape(text, Locale.getDefault(), Type.NORMAL). The same raw field print is at the seven other sites of the nodes the standard parser builds: QuotedFieldQueryNode.java:39, FuzzyQueryNode.java:55, AbstractRangeQueryNode.java:166 (sb.append(getField()).append(":")), PointQueryNode.java:89, WildcardQueryNode.java:47, RegexpQueryNode.java:93 and IntervalQueryNode.java:49. Where a node escapes its text at all, only the term text meets the escaper: WildcardQueryNode, RegexpQueryNode and IntervalQueryNode print their text raw. Two more nodes print the field raw, AnyQueryNode.java:121 and ProximityQueryNode.java:186, but no parser code builds them, and their ANY / WITHIN output does not parse back even for a plain field (f:(( a b ) WITHIN 3) parses to (f:a f:b) f:WITHIN f:3, executed).

The parser keeps the escapes: FieldName() "strips escape characters from field names" (StandardSyntaxParser.jj:336-344) and returns an UnescapedCharSequence that records which characters were escaped. They are lost only when printing. EscapeQuerySyntax.Type has only STRING and NORMAL, no field mode.

A field name must lex as one TERM token (StandardSyntaxParser.jj:328). A character outside _TERM_START_CHAR (:131-133) cannot start it, and a space, :, ", (, [, @ and the others also outside _TERM_CHAR (:134) end it (+ and - break a name only in first position), and a whole name that is a number (NUMBER, :160), the interval prefix fn: (:147) or AND/OR/NOT (:144-146) lexes as something else.

input to the syntax parser                 toQueryString prints   parsing the printed text gives
f:x (control)                              f:x                    f:x (same)
my\ field:x                                my field:x             df:my OR field:x
a\:b:x                                     a:b:x                  syntax error
my\ field:"p q"                            my field:"p q"         df:my OR the quoted query on field
my\ field:xyz~1                            my field:xyz~1.0       df:my OR field:xyz~1
my\ field:[a TO b]  (main only)            my field:[a TO b]      df:my OR a range on field
my\ field:/x/                              my field:/x/           df:my OR a regexp on field
a\\b:c (the field a\b)                     a\b:c                  ab:c (the field ab)
new FieldQueryNode("dc:title","hugo",0,0)  dc:title:hugo          syntax error
\2024:x, \fn:x, \OR:x                      2024:x, fn:x, OR:x     errors

Exhaustive check on the stock jar: of 9,119 printable-ASCII field names of length 1 or 2 (all but *; each in a hand-built FieldQueryNode(name, "x"), printed and parsed again), 3,384 fail (479 parse to a different query, 2,905 do not parse; the finder's 3,388 include 4 point-field artifacts of the harness). Over every query string of length <= 5 from a 20-symbol alphabet (163,434 parsed), 3,832 of the 6,168 round-trip failures are this bug alone.

• Who. Trees from the public getSyntaxParser() (QueryParserHelper.java:172), processed trees (which can also hold post-analysis nodes such as TokenizedPhraseQueryNode, marked not re-parseable), or hand-built nodes; the node package says the nodes "are designed to be used by UI or other text parsers" (core/nodes/package-info.java:28).
• Trigger. A field name with a syntax character, or one that is a token by itself. Namespaced names like dc:title occur in practice (#1349 / LUCENE-271 reported one in 2004, for the classic API).
• Effect. The printed text fails to parse or, silently, queries other fields. No index damage; the Query from parse is right.
• In-tree. The output only feeds exception messages (QueryTreeBuilder.java:161, BooleanQueryNodeBuilder.java:72, MatchAllDocsQueryNodeBuilder.java:42, MatchNoDocsQueryNodeBuilder.java:42, AllowLeadingWildcardProcessor.java:74) and IntervalQueryNode.toString; nothing in Lucene parses it again.
• Likelihood. Rare: the documented searches found no report in the 17 years since 2.9.0.

Oracle: parse(q) must equal parse(toQueryString(syntax tree of q)), which is what the javadoc promises. LuceneTestCase is only the base class; single thread, deterministic (same result with seeds DEADBEEF and 5EED1234).

Regression test: this same class (or its one method added to TestQPHelper); it passes with the recommended fix.

cp $REPRO/TestToQueryStringFieldNames.java $LUCENE/lucene/queryparser/src/test/org/apache/lucene/queryparser/flexible/standard/
cd $LUCENE && ./gradlew :lucene:queryparser:test --tests TestToQueryStringFieldNames -Ptests.seed=DEADBEEF -Ptests.asserts=true
# assertions off fails the same way: -Ptests.asserts=false

# gradle, stock 645d4fe14a20, -Ptests.seed=DEADBEEF -Ptests.asserts=true (stack trimmed; -Ptests.asserts=false identical)
   >     java.lang.AssertionError: printed: my field:x expected:<my field:x> but was:<df:my field:x>
   >         at __randomizedtesting.SeedInfo.seed([DEADBEEF:2DCFAF3042D63897]:0)
   >         at junit@4.13.2/org.junit.Assert.assertEquals(Assert.java:120)
   >         at org.apache.lucene.queryparser.flexible.standard.TestToQueryStringFieldNames.testFieldNameWithSpace(TestToQueryStringFieldNames.java:30)

:lucene:queryparser:test (FAILURE): 1 test, 1  failure

# standalone runner, seed DEADBEEF: main 645d4fe14a20 and 10.6.0 fail with -ea and -da, same message

• Contract. QueryNode.java:26: "convert to a query string understood by the query parser". core/nodes/package-info.java:42-43: "A query node tree can also generate a query string that can be parsed back by the original text parser, at this point only the standard lucene syntax is supported." EscapeQuerySyntax.java:22-23: the escaper exists "to allow the QueryNode to escape the queries, when the toQueryString method is called". The colon case does not even parse.
• Deliberate input. StandardSyntaxParser.jj:337; TestQPHelper.java:817-820 asserts a\-b:c, a\+b:c, a\:b:c and a\\b:c give the fields a-b, a+b, a:b and a\b; LUCENE-161 (#1239) added escaped field names in 2004.
• Precedent. CHANGES.txt:296-298 (11.0.0 Bug Fixes): "GITHUB#7865 AbstractRangeQueryNode#toQueryString(EscapeQuerySyntax) now returns a string which can be parsed back into the original node." Field escaping never came up in #14697 (its tests use an ordinary field name).
• No intent found. In the material reviewed (the flexible parser's source, tests and javadoc, CHANGES and MIGRATE, the six commits that added the eight sites, the review threads in the follow-up) nothing says fields are printed raw on purpose. The three core lines are unchanged since 2009; the later sites (2010-2025) copied the pattern without comment.

The strongest case for "intended": a 2017 java-user reply on range round trips said "There has never been a guarantee that going back and forth between a parsed query and its string representation is idempotent"; open #8667 calls toQueryString a design error; #1349 / LUCENE-271, the same symptom for the classic Query.toString(field) (dc\:title:hugo printed as dc:title:hugo), was closed "Not A Problem" in 2011; inside Lucene the output feeds only error messages and one toString; some nodes say their output is not re-parseable (TokenizedPhraseQueryNode.java:47; MultiPhraseQueryNode prints [MTP[...]]); and the escaper never had a field mode.

Why it fails: Query.toString is documented as user-readable, while QueryNode#toQueryString and its package promise text the parser understands and can parse back. In the same 2017 thread Michael McCandless answered "Why don't we fix this in Lucene?", and #14697 then fixed the range round trip as a bug, its approver noting "we're changing behaviour that is public and has been around a long time, but it's also a bug fix". #8667 is about non-standard syntaxes; this fails with the standard one. The not-re-parseable nodes are post-analysis nodes and are marked as such; the syntax parser's field nodes carry no such mark. Being used only in messages limits impact, not intent. For hand-built nodes the promise can hold only for names the standard syntax can write at all (an empty name cannot be written); the claim is about names it can write, as the parser's own escaped-field tests show. A read-only cross-model refute review returned only a procedural "not real" (it could not run Java), agreed on the root cause and rated this argument weak; the intended-lens review said real.

One fix was written and validated in a scratch worktree of 645d4fe14a20; the alternatives are described, and one partial variant was run by a verifier. The output of toQueryString changes for such names, so maintainers may want it in 11.0 only, as for #14697.

+43/-8 in 9 files (tidy-formatted). QueryNodeImpl gets protected getFieldEscaped(CharSequence, EscapeQuerySyntax); FieldQueryNode, QuotedFieldQueryNode, FuzzyQueryNode, AbstractRangeQueryNode, PointQueryNode, WildcardQueryNode, RegexpQueryNode and IntervalQueryNode call it instead of printing the raw field. AnyQueryNode.java:121 and ProximityQueryNode.java:186 stay as they are: no parser builds those nodes, and their ANY / WITHIN text does not parse back even for a plain field, so escaping their field alone would not make them round-trip (with the fix their output is unchanged, executed); a maintainer may still want them to call the helper for consistency. NUMBER_FIELD is Pattern.compile("[0-9]+(\\.[0-9]+)?"), the grammar's NUMBER token.

protected CharSequence getFieldEscaped(CharSequence fld, EscapeQuerySyntax escaper) {
  CharSequence escaped = escaper.escape(fld, Locale.ROOT, EscapeQuerySyntax.Type.NORMAL);
  // a field name must lex as a term; a bare number or "fn" would lex as a number or as the
  // interval function prefix "fn:", so escape its first character
  if (escaped != null
      && ("fn".contentEquals(escaped) || NUMBER_FIELD.matcher(escaped).matches())) {
    return "\\" + escaped;
  }
  return escaped;
}

Trade-offs

• Output change: names with syntax characters print with backslashes (my\ field:x); names equal (ignoring case) to AND, OR, NOT, TO, WITHIN, SENTENCE, PARAGRAPH or INORDER, number-shaped names (2024, 1.5) and fn get a leading one (\to:x, \2024:x); plain names print as before. Worth a CHANGES entry.
• Inherits the escaper's own gaps. Sibling LFV-398: @ after the first character and the names && / || still fail until EscapeQuerySyntaxImpl is fixed; in the exhaustive printable-ASCII check those are the only real field-name failures left. Beyond ASCII, escapeIgnoringCase matches on a lower-cased copy and inserts the backslashes at its indexes, so a character whose lower case is longer shifts them: with the fix the field İ:a (U+0130) prints İ:\a:x, which does not parse, and İ a prints İ \a:x, which parses to df:İ a:x (executed; stock fails too). Open outside PR #16853 fixes that escaper bug.
• Reusing the term escaper is safe for fields: \to parses back to the field to, and * / ? are escaped only when they were escaped in the input.
• Puts two standard-syntax rules (number, fn:) in the core node base class, which already prints standard syntax (:, ~, [ TO ]).

What the checks validated

Gradle --no-daemon in a scratch worktree of 645d4fe14a20, OpenJDK 25.0.3 (2026-10-31). Stock: the repro fails with assertions on and off. With the fix:

• the repro passes with assertions on and off;
• the whole :lucene:queryparser:test passes: 534 tests (533 plus the repro), 0 failures, 1 skipped, seeds DEADBEEF and CAFEBABE; checkGoogleJavaFormat, forbiddenApisMain and ecjLintMain pass;
• the finder's 4-test class, the independent reproduction's 11-test class and the intended-behaviour check's 9-test class pass on the fixed sources;
• the exhaustive harness on the fixed sources: field mode 3,388 -> 101 of 9,119 (4 point-field artifacts of the harness; 95 names with @ after the first character, && and ||, all LFV-398); syntax mode 6,168 -> 2,144 of 163,434, all LFV-398 classes. Without the number/fn rule 212 remain (110 all-digit names and fn more). The field mode builds FieldQueryNode(name, "x") directly, so it covers that node; the other sites were run by the probes and the verifiers' tests.

An independent fact check re-ran all of this (gradle stock and fixed, -ea/-da, the module, lint, the harness) with the same results.

Add EscapeQuerySyntax.Type.FIELD and let EscapeQuerySyntaxImpl own the field rules (the term rules plus number, fn, inner @, &&, ||); the nodes call escape(field, locale, Type.FIELD).

Trade-offs

Keeps the syntax rules in the syntax's escaper, but adds a constant to a public enum that custom EscapeQuerySyntax implementations must handle.

When the field is an UnescapedCharSequence (as FieldName() returns), print toStringEscaped().

Trade-offs

Reproduces what the user typed, but does nothing for hand-built nodes or String field names (new FieldQueryNode("my field", ...)), nor for parsed interval queries, whose node gets field.toString() and loses the escape flags (StandardSyntaxParser.jj:379); it is incomplete on its own.

Change only FieldQueryNode.toQueryString.

Trade-offs

Fixes the term, colon and hand-built-node cases; phrase, fuzzy and range still fail (3 of 11 tests), because each node prints the field itself.

Method: semi-formal (bounded-exhaustive translation validation on the real jar).

• Finder. Hunt unit on the flexible, simple and XML query parsers (2026-10-26, main ea57360912ed). FlexRoundTrip mode field: hand-built FieldQueryNode(name, "x") for 9,119 names, printed and parsed again, 3,388 mismatches (3,384 genuine, 4 point-field artifacts of the harness); mode syntax: every string of length <= 5 over a 20-symbol alphabet, 163,434 parsed, 6,168 failures, 3,832 of them this class alone and 264 more combined with @. Then a 4-test JUnit on the stock jars (3 of 4 fail on main with -ea and -da, and on 10.5.2).
• Verification. Read-only cross-model reviews (OpenAI GPT-5.6), an independent reproducer, an intended-behaviour devil's advocate, an adjudication, the version-history runs, an audit and a one-test minimal repro (see the checks below).