Classic and flexible query parsers: a tab, line break or ideographic space touching a range bound becomes part of the bound

Both query grammars, the classic QueryParser.jj and the flexible StandardSyntaxParser.jj, declare U+0020, TAB, LF, CR and U+3000 as whitespace and skip it in the Range lexical state as well. But the token for a range bound, RANGE_GOOP, is any run of characters other than U+0020, ] and }. The lexer keeps the longest match, so a TAB, LF, CR or U+3000 that touches a bound is swallowed into it.

Two outcomes follow. When the character separates a bound from TO, the bound and TO become one token and the parse fails: date:[20200101<TAB>TO<TAB>20201231] throws in both parsers. When it is glued to a bound but not to TO (right after [/{, right before ]/}, or between a bound and the space next to TO), the query parses without error and the bound carries the character: over five dates date:[<LF>20200101 TO 20201231<LF>] matches 4 instead of 3, date:{20200101 TO 20201231<TAB>} admits its own exclusive endpoint, date:[<U+3000>20200101 TO 20201231] matches nothing, and date:[20200101<LF> TO 20201231], a value pasted with a trailing newline, matches 2 instead of 3. It affects the classic parser and its subclasses, StandardQueryParser and PrecedenceQueryParser with default parser settings, and every release since the TO syntax arrived in 1.3-rc1. The fix adds the four characters to the exclusion list in both grammars and regenerates the token managers.

• Grammar whitespace: <#_WHITESPACE: ( " " | "\t" | "\n" | "\r" | "\u3000") > (QueryParser.jj:187; the same at StandardSyntaxParser.jj:135). Outside ranges these five characters are also excluded from terms (_TERM_START_CHAR, QueryParser.jj:183-185), so there they always separate clauses: upstream testTabNewlineCarriageReturn and testCJK assert it.
• Range state: after [/{ the lexer uses the rules RANGE_TO (TO), RANGEIN_END (]), RANGEEX_END (}), RANGE_QUOTED and RANGE_GOOP (QueryParser.jj:221-227; StandardSyntaxParser.jj:195-201), and the whitespace SKIP rule <DEFAULT, Range> SKIP : { < <_WHITESPACE>> } (QueryParser.jj:191-193; <DEFAULT, Range, Function> at StandardSyntaxParser.jj:139-141).
• Range rule: bound, TO, bound, closing bracket (QueryParser.jj:343-347; StandardSyntaxParser.jj:855-859).

1. RANGE_GOOP is (~[ " ", "]", "}" ])+ (QueryParser.jj:226, StandardSyntaxParser.jj:200): it excludes only U+0020 of the five whitespace characters.
2. At a TAB that starts a token and is followed by a letter or digit, SKIP matches 1 character and RANGE_GOOP matches more, so the TAB joins the bound. SKIP wins only a tie, when the character starts a token and is followed by U+0020, ], } or the end of the input. Inside a bound there is no token start: RANGE_GOOP runs on through the character, so one glued to the end of a bound is swallowed whatever follows it (20200101<LF> TO gives the bound 20200101<LF>).
3. Loud: in [20200101<TAB>TO<TAB>20201231] the whole inside is one RANGE_GOOP, the parser meets ] where it needs TO, and throws (Was expecting: "TO" in the classic parser, INVALID_SYNTAX_CANNOT_PARSE in the flexible one).
4. Silent: a character glued to a bound but not to TO (right after [/{, right before ]/}, or between a bound and the space next to TO) stays in the bound text. The classic parser passes it to newRangeQuery, where Analyzer.normalize (QueryParserBase.java:550, :556) only lowercases for StandardAnalyzer; the flexible parser builds its bound nodes from the token text the same way (StandardSyntaxParser.jj:870-873). The result is a TermRangeQuery whose bound contains the character. The open-bound check "*".equals(goop1.image) (QueryParser.jj:354) also fails, so [<LF>* TO b] searches from the literal term <LF>*.

Five StringField dates 20191231, 20200101, 20200615, 20201231, 20210101; both parsers give the same result on every line (the fact check ran four more parsers built on the two grammars, with the same result).

• Who is exposed. Applications that parse query text with the classic QueryParser or its subclasses MultiFieldQueryParser, ComplexPhraseQueryParser and ExtendableQueryParser, with StandardQueryParser, or with PrecedenceQueryParser (which extends it), with default parser settings (the caller supplies the analyzer; the repro passes StandardAnalyzer, and the defect is in the lexer, before any analysis). Executed for QueryParser, MultiFieldQueryParser, StandardQueryParser and PrecedenceQueryParser; the other two inherit the classic grammar. Only these two grammars have a range syntax: the surround parser (its grammar has no Range state) and SimpleQueryParser (no range operator) are not affected.
• Inputs that reach it. A range typed over several lines or with tabs in a query text area; a value with a trailing newline or tab pasted between the brackets; Windows line endings (<CR><LF> breaks even indented input); the full-width space U+3000 that CJK input methods type, the case LUCENE-1528 added U+3000 for; generated query strings built from templates.
• Loud vs silent. A character that touches TO makes TO part of a bound token and the parse fails: a ParseException for a range whose parts are separated by one of these characters, and quoting the bounds does not help. A character glued to a bound but not to TO silently changes that bound (extra hits, an exclusive endpoint admitted, or no hits at all): right after [/{, right before ]/}, a value pasted with a trailing newline (date:[20200101<LF> TO 20201231]: 2 hits instead of 3) or a tab typed after TO (date:[20200101 TO <TAB>20201231]: 0 instead of 3). A character with a space or bracket on both sides is skipped.
• Scale. One query string reproduces it; no setting, index size or concurrency is involved, and assertions on or off behave the same.
• Setup after lexing. The hit counts are for StringField terms and StandardAnalyzer, whose normalize only lowercases; an analyzer whose normalization strips or maps these characters would hide part of the silent outcome (not run). Both parsers first try to read each bound as a date in their locale, but with no date resolution configured (the default, as in the repro) DateTools.dateToString throws on the null resolution and both keep the bound text, so the random test locale cannot change the results; with a date resolution set, a bound the locale's date format reads as a date is rewritten first (not run).
• How likely. Rare: it needs one of these characters inside the brackets of a range, the parse fails whenever one touches TO, and the searches (see Prior reports) found no public report in over 20 years of releases.
• Downstream. Solr's fork of the classic grammar has the same RANGE_GOOP line and Range-state SKIP (read on apache/solr main by the finder).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<date:[20200101 TO 20201231]> but was:<date:[
20200101 TO 20201231
]>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:120)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.queryparser.classic.TestRangeBoundWhitespace.testLineBreakInsideRangeBrackets(TestRangeBoundWhitespace.java:27)

• The grammar's own rules. _WHITESPACE lists U+0020, TAB, LF, CR and U+3000 and is skipped in the Range state (QueryParser.jj:187, :191-193; StandardSyntaxParser.jj:135, :139-141). Because RANGE_GOOP swallows the other four, that SKIP fires for them only in a tie, so the same character is skipped or kept depending on its neighbours: date:[<TAB> 20200101 ... gives the lower bound 20200101, date:[<TAB>20200101 ... gives <TAB>20200101. No design note, test or discussion found explains this distinction.
• Tests. testTabNewlineCarriageReturn (QueryParserTestBase.java:896, TestQPHelper.java:937) asserts that "+weltbank\n+worlbank", "+weltbank\r\n+worlbank" and similar equal "+weltbank +worlbank"; testCJK (QueryParserTestBase.java:250-255, TestQPHelper.java:324-329) that "term\u3000term\u3000term" equals the space form. No test has a range with a raw TAB, LF, CR or U+3000; with the fix all 30 upstream queryparser suites pass.
• History. The 2002 commit that added the TO syntax also extended the whitespace SKIP, then " " and TAB, to the new range states, and defined a bound token that excludes only " ". Later the same day (about 17 hours after the commit), its author wrote on dev@: "right now, we're using whitespace as delimiters in the range -- which means that dates like 2002-01-01 1:35 PM are no good unless enclosed in quotes". The 2004 change (LUCENE-93) "treats \n and \r characters in query strings the same way that \t and " " are treated"; LUCENE-1528 (2009) added U+3000 for CJK text. Neither touched the bound token. The LUCENE-1528 review asked how to avoid "defining the whitespace chars in two places" and missed this third copy.
• Committer remark. In the merged test PR #13323, dweiss wrote that range terms "are parsed differently compared to normal tokens. This is strange but I bet there was a reason for it - now long forgotten. I think, logically, it should accept the same term notation that is used elsewhere" (said about escapes).

The strongest case for "intended" (built by the devil's-advocate verifier; the cross-model reviews rated it weak, the cross-model fact check of this doc moderate): the delimiter has always been written " TO " with ASCII spaces (the 2002 commit message, the LUCENE-7821 CHANGES entry, the test comment // " TO " is required between range endpoints at QueryParserTestBase.java:637); the user docs show only space-separated ranges; the package docs call the parser "designed for human-entered text, not for program-generated text" and say date ranges "are better added directly through the query API"; the behaviour is over 20 years old with no report found; and today a raw TAB inside an unquoted bound (k:[a<TAB>b TO c]) parses, which the fix would break.

Why it fails: " TO " is about the keyword being required and delimited (LUCENE-7821 was about ranges without TO and never discusses the kind of whitespace). The docs' silence is filled by the grammar's own declarations and the tests. "Human-entered text" argues for the fix: people press Enter in a text area, and CJK input methods type U+3000. Age shows the case was never exercised, not that it was chosen. The raw TAB inside an unquoted bound is the real compatibility question: no doc, test or design note mentions it, the 2002 author named quoting for a date containing spaces, and quoting keeps working with the fix; maintainers should weigh it against consistent Range-state whitespace.

Option A is the fix the finder proposed and this doc validated with gradle. B merges it with the escape fix of #13234; C and D are listed because a reviewer may ask about them. Add the new test (or its cases in QueryParserTestBase) with whichever is chosen.

One line in each grammar (QueryParser.jj:226, StandardSyntaxParser.jj:200), then ./gradlew :lucene:queryparser:regenerateClassicQueryParser :lucene:queryparser:regenerateFlexibleStandardSyntaxParser, which changes 7 lines in each generated token manager and updates the two checksum files.

-| <RANGE_GOOP:   (~[ " ", "]", "}" ])+ >
+| <RANGE_GOOP:   (~[ " ", "\t", "\n", "\r", "\u3000", "]", "}" ])+ >

Trade-offs

• An unquoted bound that contains one of the four characters stops parsing: k:[a<TAB>b TO c] gives the bounds a<TAB>b and c today and a ParseException with the fix, while k:["a<TAB>b" TO c] keeps working. The 2002 author described whitespace as the range delimiter and said a date containing spaces needed quotes; propose it for main / 11.0 with a CHANGES entry ("the query parser syntax may change from release to release", classic package-info.java:74-75).
• A JavaCC character list holds characters only and cannot name _WHITESPACE, so the whitespace list is now written three times per grammar. Add a comment that ties _WHITESPACE, _TERM_START_CHAR and RANGE_GOOP together (the LUCENE-1528 review already worried about two copies), and keep them in step if _WHITESPACE ever grows.
• Nothing else changes: the rule differs only on the four characters and only in the Range state, so only ranges that contain one of them parse differently; with the fix quoted bounds next to a TAB parse too.

What the checks validated

• Scratch worktree at a9779ef3c193, gradle, JDK 25: the new test fails 5 of 6 on stock (assertions on and off) and passes 6 of 6 with the fix; the whole :lucene:queryparser:test passes with assertions on (seed 5EED0845) and off (seed 5EED0846): 31 suites, 539 tests, 0 failures, 1 skipped (the 30 upstream suites include TestQueryParser, TestStandardQP, TestQPHelper, TestMultiFieldQueryParser, TestPrecedenceQueryParser, TestComplexPhraseQuery and TestExtendableQueryParser). checkGoogleJavaFormat, both regenerate checksum checks and forbiddenApisMain/forbiddenApisTest pass.
• After the fact checks, the test file exactly as printed (the formatter's one-line assertEquals) was run again in a fresh scratch worktree: checkGoogleJavaFormat and forbiddenApisTest pass on stock and with the fix (with both regenerate checksum checks); the test fails 5 of 6 on stock and passes 6 of 6 with the fix, assertions on and off.
• A probe with the regenerated token managers gives every input the bounds of its space form in both parsers; <CR><LF> input and quoted bounds next to a TAB parse; [<LF>* TO b] is open-ended again.
• Earlier, with the finder's regenerated token managers, the finder's 4 tests and the independent reproducer's 7 tests passed, the bounded lexer check held in all four lexer/state pairs, and 14 upstream queryparser test classes (323 tests) passed.

For example (~["\\", " ", "\t", "\n", "\r", "\u3000", "]", "}"] | "\\" ~[])+: the rule proposed in closed PR #13887 plus the four characters. It would close #13234 and this bug in one change.

Trade-offs

It also changes how escapes inside bounds behave. #13887 proposed exactly that and was closed by its author (2026-03-04) without a review or a maintainer comment; dweiss noted on #13323 that such a change "will break backward compatibility but maybe consistency is worth this cost", so it needs that discussion first. #13887's rule alone leaves this bug in place.

Strip the four characters from both ends of each bound before the * check and getRangeQuery.

Trade-offs

Fixes the silent cases (a character at either end of a bound) if it runs before the * check. 20200101<TAB>TO<TAB>20201231 is still one token, so the ParseException stays; and the lexer and the grammar's SKIP rule keep disagreeing.

Say so in the classic package docs and the StandardQueryParser javadoc. Not run.

Trade-offs

Changes no output, leaves the silent bound changes, and contradicts the grammar's own Range-state SKIP.

Method: static comparison of the grammars' lexer tables, then bounded-exhaustive checking of the generated token managers (the study's "semi-formal" tier; the finder tagged it formal). A hunt unit, query-parser whitespace tables versus Unicode White_Space, 2026-10-28.

• Table diff. A custom program (not a solver) extracted the character classes from QueryParser.jj and StandardSyntaxParser.jj and checked set obligations between them. Obligation O3, "the grammar's _WHITESPACE is a subset of the RANGE_GOOP exclusions", was violated for U+0009, U+000A, U+000D and U+3000 in both grammars. The signal came from this leg.
• Lexer check. All 111,150 inputs of length 1 to 4 over an 18-symbol alphabet, run through the real generated token managers per lexer and lexical state: the grammar's own whitespace ends up inside a token only in the Range state and only in RANGE_GOOP; the DEFAULT state is clean. With the regenerated (fixed) token managers the property holds in all four lexer/state pairs.
• Parser sweep. A metamorphic sweep (replace a space by each whitespace character and compare the parsed query) found 720 mismatches in 2,358 hard-oracle checks, all in the 10 range templates.
• Then a public-API JUnit on the stock main jars, followed by independent reproduction, cross-model reviews, a devil's advocate, an adjudication and an audit (see the checks).