Flexible query parser: toQueryString prints fn:fuzzyTerm(abc) as fn:fuzzyTerm(abc 2128), which does not parse back

The flexible query parser documents that a query node tree "can also generate a query string that can be parsed back by the original text parser", and QueryNode#toQueryString is "convert to a query string understood by the query parser". For interval functions this fails. IntervalQueryNode#toQueryString prints the field, a colon and the function's toString(), and three of those methods print text the parser rejects or reads differently: FuzzyTerm runs maxEdits and maxExpansions together (fn:fuzzyTerm(abc) prints as fn:fuzzyTerm(abc 2128), which re-parses as maxEdits 2128 and throws), Wildcard prints maxExpansions:5 where the grammar takes a bare number, and AnalyzedText quotes a literal only when Java's \s matches, so literals with : or a keyword print bare (syntax error) and some re-parse with no error as a different query (U+3000 splits a phrase into two words; a ) inside a literal closes the function early). The Query that StandardQueryParser.parse builds is unaffected; applications that round-trip node text and Lucene's own exception messages get the wrong text. Every release from 9.1.0 to 10.6.0 and main; a fix of +12/-4 lines in the three printers passes the repro and the whole queryparser module.

• IntervalQueryNode#toQueryString is String.format(Locale.ROOT, "%s:%s", field, source) (IntervalQueryNode.java:47-50): the field, a colon and the function's toString(). It ignores its EscapeQuerySyntax argument, and IntervalQueryNode#toString delegates to it (:52-55), so there is no separate display form.
• FuzzyTerm.toString (FuzzyTerm.java:52-60) formats "fn:fuzzyTerm(%s %d%s)" with the term, maxEdits and the Integer maxExpansions: no space before the second number, so 2 and 128 print as 2128 and 1 and 5 as 15. The grammar reads fn:fuzzyTerm(abc 2128) as term abc with maxEdits 2128 (StandardSyntaxParser.jj:659-679), and building the query throws "max edits must be 0..2, inclusive" (FuzzyAutomatonBuilder.java:38-39, the throw at :39).
• Wildcard.toString (Wildcard.java:44-51) prints the raw glob and " maxExpansions:" + maxExpansions. The grammar takes the glob as a term, number or quoted token followed by an optional bare number (StandardSyntaxParser.jj:641-657), so maxExpansions:5 is a syntax error, and a quoted glob such as "a b*" loses its quotes.
• AnalyzedText.requiresQuotes (AnalyzedText.java:53-55) quotes a literal only if Pattern.compile("[\\s]") finds a match. The lexer is stricter: a term cannot start with whitespace (including U+3000) or any of + - ! ( ) : ^ @ < > = [ ] " { } ~ \ / (StandardSyntaxParser.jj:131-133) and ends at any of them except + and - (:134); it reads AND/&&, OR/|| and NOT/! as operators (:144-146) and treats U+3000 as whitespace (:135), which Java's \s does not match by default. So most such literals print bare and fail to parse, some come back as a different query, and a literal such as "x-y" still round-trips. FuzzyTerm reuses this check for its term.

Field body, executed on stock builds with assertions on and off:

Also executed, with assertions off: fn:or("a)(b" c) prints as body:fn:or(a)(b c) and parses back with no error as body:a (body:b body:c) instead of body:or(BLOCK(a,b),c); fn:or("x-y" z) round-trips; and the upstream test's own input fn:fuzzyTerm(dfe 2 128) (TestStandardQPEnhancements.java:161) prints as body:fn:fuzzyTerm(dfe 2128) and fails to re-parse.

Over a grid of 8,352 interval expressions (18 leaves, including quoted literals with :, AND, &&, || and U+3000, and every depth-1 function over them), 6,100 do not round-trip on main, 121 of them as a silently different query; with the recommended fix, none. The grid is a test design, not a usage frequency.

StandardQueryParser.parse builds the Query from the tree and never re-parses this text. In-tree, toQueryString output appears only in exception messages (QueryTreeBuilder.java:161, BooleanQueryNodeBuilder.java:72, MatchAllDocsQueryNodeBuilder.java:42, MatchNoDocsQueryNodeBuilder.java:42, AllowLeadingWildcardProcessor.java:74). For example, with a clause limit of 2, parse("a b fn:fuzzyTerm(abc 1 2)", "body") throws "Too many boolean clauses, the maximum supported is 2: body:a body:b body:fn:fuzzyTerm(abc 12)".

• Who. Applications that use the flexible parser's node tree as an intermediate form: parse user text with StandardSyntaxParser, inspect or rewrite the tree, then print it with toQueryString and parse, store, cache or forward the text. Also anyone reading logs or exception messages that print interval nodes: fn:fuzzyTerm(abc 12) hides the real maxEdits 1 and maxExpansions 2.
• Default vs opt-in. Default parser configuration and public API; the trigger needs the fn: interval syntax and an application that re-parses node text (no in-tree caller does).
• Effect. An exception on the second parse, or, for some quoted literals, a different query with no error: with U+3000 a two-word phrase becomes two separate words, and fn:or("a)(b" c) comes back as body:a (body:b body:c). The results of the original parse() are unaffected.
• Scale. Any index; deterministic string formatting.
• How likely: rare. It needs both the niche fn: syntax and a re-parse of toQueryString text; the documented searches found no prior public report since 9.1.0 (2022-03). Two outside ports of the parser copy FuzzyTerm.toString unchanged; no report was found there either.

The test prints the node tree of fn:fuzzyTerm(abc) on field f and parses the printed text. The try/catch only puts the printed string into the failure message (the cause is kept); with a fix, the final assertEquals checks that the re-parsed query equals the direct parse. It covers the fuzzyTerm part, the least arguable one; the wildcard and literal cases are shown in the table above and were executed by the hunt's and a verifier's tests.

Regression test: this same class; it passes with the recommended fix.

cp $REPRO/TestFuzzyTermToQueryString.java $LUCENE/lucene/queryparser/src/test/org/apache/lucene/queryparser/flexible/standard/
cd $LUCENE && ./gradlew :lucene:queryparser:test --tests TestFuzzyTermToQueryString -Ptests.seed=DEADBEEF -Ptests.asserts=true
# assertions off fails the same way: -Ptests.asserts=false

# gradle, stock 645d4fe14a20, -Ptests.seed=DEADBEEF -Ptests.asserts=true (stack trimmed; -Ptests.asserts=false fails the same way in the standalone runner)
> Task :lucene:queryparser:test FAILED

TestFuzzyTermToQueryString > testToQueryStringParsesBack FAILED
    java.lang.AssertionError: expected toQueryString output that parses back, but 'f:fn:fuzzyTerm(abc 2128)' fails: java.lang.IllegalArgumentException: max edits must be 0..2, inclusive; got: 2128
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:D6B50425F7113193]:0)
        at org.apache.lucene.queryparser.flexible.standard.TestFuzzyTermToQueryString.testToQueryStringParsesBack(TestFuzzyTermToQueryString.java:38)

        Caused by:
        java.lang.IllegalArgumentException: max edits must be 0..2, inclusive; got: 2128
            at org.apache.lucene.core@11.0.0-SNAPSHOT/org.apache.lucene.search.FuzzyAutomatonBuilder.<init>(FuzzyAutomatonBuilder.java:39)
            at org.apache.lucene.core@11.0.0-SNAPSHOT/org.apache.lucene.search.FuzzyQuery.getFuzzyAutomaton(FuzzyQuery.java:191)

# standalone runner, seed DEADBEEF: main 645d4fe14a20 and 10.6.0 fail with -ea and -da, 9.1.0 fails with -ea

• The contract: QueryNode.java:26 ("convert to a query string understood by the query parser") and core/nodes/package-info.java:42-43 ("A query node tree can also generate a query string that can be parsed back by the original text parser, at this point only the standard lucene syntax is supported"); EscapeQuerySyntax exists "to allow the QueryNode to escape the queries, when the toQueryString method is called" (EscapeQuerySyntax.java:22-23).
• The inputs are accepted, tested syntax: CHANGES 9.1.0 lists LUCENE-10223 (interval functions) and LUCENE-10414 ("Add fn:fuzzyTerm interval function to flexible query parser", lucene/CHANGES.txt:4415-4417, :4452-4453); upstream tests parse fn:wildcard(foo* 128) ("Explicit maxExpansions") and fn:fuzzyTerm(dfe 2 128) (TestStandardQPEnhancements.java:97-102, :156-162); quoted literals are documented ("Quoted or unquoted character sequences ...", intervalfn/package-info.java:149). The package docs list only fn:wildcard(glob) and not fn:fuzzyTerm, so "accepted, tested syntax" is the accurate wording.
• The code aims for parseable output: on inspection, the other interval functions print their grammar syntax (the controls round-trip and the 8,352-expression grid shows no failure outside the three printers; there is no dedicated round-trip suite over all functions) and AnalyzedText already quotes when it thinks it must.
• Precedent: "GITHUB#7865 AbstractRangeQueryNode#toQueryString(EscapeQuerySyntax) now returns a string which can be parsed back into the original node." (lucene/CHANGES.txt:296-298, 11.0.0 Bug Fixes; PR #14697 with regression tests).

The strongest case for "display only": IntervalFunction#toString has no javadoc; the only upstream test that touches the output prints it "for visual diagnostics" and asserts nothing (TestStandardQPEnhancements.java:180-184); maxExpansions:N looks like a deliberate human-readable label; a 2017 java-user reply said a round trip between a parsed query and its string form was never guaranteed; open #8667 calls the toQueryString design flawed; and no in-tree caller re-parses the text.

Why it fails: IntervalQueryNode#toQueryString, the method whose javadoc promises parser syntax, returns exactly these strings, and IntervalQueryNode#toString delegates to it, so even if IntervalFunction#toString were diagnostic only, toQueryString would still need the fix. The 2017 reply was answered in the same thread with "Why don't we fix this in Lucene?", and the project later fixed the range-node case as a bug (GITHUB#7865). #8667 is about nodes built by custom syntax parsers, while the package docs promise the round trip for "the standard lucene syntax". And abc 2128 cannot be a deliberate display choice. All verifiers and the adjudication rejected the intended-behaviour case.

Two fixes were written and validated in a scratch worktree of 645d4fe14a20: the full fix (recommended) and a smaller numbers-only fix; a dedicated serializer is described but not written.

The change touches FuzzyTerm.java, Wildcard.java and AnalyzedText.java (the hunt's fix sketch, reformatted by applyGoogleJavaFormat). fn:fuzzyTerm prints both numbers separated (fn:fuzzyTerm(abc 2 128)); fn:wildcard prints the bare number and quotes a glob that needs it; a literal is quoted when it is empty, contains whitespace including U+3000, contains any character a term cannot start with, equals AND, OR, NOT, && or ||, or starts with fn: (redundant, since : is already covered). That covers every literal the lexer would not read back as one term, and a few it would: + and - may follow the first character of a term, so fn:or(a-b c) now prints as fn:or("a-b" c). Quoting more than needed changes the text but not its meaning: a quoted and an unquoted literal build the same node. Apart from such extra quotes, every other printed form is unchanged.

// FuzzyTerm.toString()
"fn:fuzzyTerm(%s %d %d)",
// Wildcard.toString()
AnalyzedText.requiresQuotes(wildcard) ? '"' + wildcard + '"' : wildcard,
maxExpansions == 0 ? "" : " " + maxExpansions);
// AnalyzedText.requiresQuotes(String term)
return term.isEmpty()
    || Pattern.compile("[\\s\\u3000+\\-!():^@<>=\\[\\]\"{}~\\\\/]").matcher(term).find()
    || term.equals("AND") || term.equals("OR") || term.equals("NOT")
    || term.equals("&&") || term.equals("||") || term.startsWith("fn:");

Trade-offs

• Changes toString only; nothing on a search path. The pattern is compiled per call, as today; a static final Pattern would be a natural cleanup.
• A hand-built node whose literal holds a raw double quote still prints unparsably (the inner quote is not escaped). The parser itself keeps backslashes inside interval literals (a separate grammar bug); when that is fixed, the printer must also escape " and \ inside quotes, so the two fixes should agree. More generally, the public constructors can build nodes the current syntax cannot represent, so the fix is validated for parser-originated input and the tested grid, not as a general serializer.
• Not a complete serializer fix: IntervalQueryNode#toQueryString still ignores its EscapeQuerySyntax argument, and the field: prefix is still printed unescaped (LFV-356, a separate fix site).

What the checks validated

Gradle in a scratch worktree of 645d4fe14a20, OpenJDK 25.0.3, seed DEADBEEF (2026-10-31). Stock: the minimal test fails, a verifier's 8-test round-trip class fails 5 of 8. With the fix:

• the minimal test passes with assertions on and off;
• the verifier's 8-test class (fuzzyTerm defaults and explicit arguments, wildcard with maxExpansions, the "a:b" and U+3000 literals, 3 controls) passes 8 of 8;
• the whole :lucene:queryparser:test passes: 30 classes, 533 tests, 0 failures, 1 skipped;
• checkGoogleJavaFormat, ecjLintMain and forbiddenApisMain for queryparser pass.

Earlier, on an identical copy of the four files: the finder's 5 tests, 132 upstream queryparser tests, 7 extra edge cases and the 8,352-expression round trip (0 failures) all pass with it.

the FuzzyTerm format "fn:fuzzyTerm(%s %d %d)" and the Wildcard suffix " " + maxExpansions; literal quoting unchanged, to be fixed together with the grammar-side unescaping.

Trade-offs

• Removes the least arguable part with no interaction with the grammar bug.
• Leaves the quoted-literal cases ("a:b", "AND", U+3000) broken until then.

What the checks validated

Same run: the minimal test passes with assertions on; the verifier's 8-test class has exactly its 2 literal tests still failing. Module suite and lint not run for this subset.

Give IntervalFunction an explicit toQueryString(EscapeQuerySyntax) and call it from IntervalQueryNode#toQueryString, leaving toString free to be a display form.

Trade-offs

• Makes the contract explicit and lets the printer use the escaper once the grammar unescapes interval literals.
• More code and a new method on every interval function class, while, on inspection, the existing toString methods already print grammar syntax for every other function.

Method: semi-formal (a bounded-exhaustive check of the stock jars, confirmed by JUnit tests).

• Signal. While writing a reference model of the interval syntax, an exploratory probe printed fn:fuzzyTerm(abc 2128) for fn:fuzzyTerm(abc).
• Exhaustive check. Every expression of the unit's grid (18 leaves, including quoted literals with :, AND, &&, || and U+3000, and every depth-1 function over them: 8,352 expressions) was parsed with StandardSyntaxParser, printed with toQueryString, parsed again with StandardQueryParser, and the two queries compared with equals.
• Confirmation. A public-API JUnit failed 4 of 5 on main with assertions on and off; then an independent reproducer, a devil's-advocate verifier, read-only cross-model reviews (OpenAI GPT-5.6) and an adjudication checked it, and a follow-up ran the version history, a prior-report search and a security screen.