RegExp no longer parses an escaped range start such as [\.-z] as a range

Write the character class [\.-z] and you mean every character from . to z: 77 of the 95 printable ASCII characters. Since Lucene 10.2.0, RegExp reads it as just three characters, ., - and z. So RegexpQuery and the classic query parser's /.../ syntax silently match other terms (3 instead of 5 in a six-term index), and a valid pattern such as [\!--] throws. Lucene 10.1.0, java.util.regex and the grammar in the RegExp javadoc all read [\.-z] as a range. The cause is a parser branch that #14227 added: it never checks for a - after an escaped character.

You want terms made of one character between . and z. Since . is a reserved character, the RegExp javadoc tells you to escape it, and to do so "also in character classes". So you write:

new RegexpQuery(new Term("f", "[\\.-z]"));   // regex text: [\.-z]

The range from . (U+002E) to z (U+007A): 77 of the 95 printable ASCII characters, including /, the digits, A-Z and a-z. It does not include - (U+002D), one code point below the dot.

Exactly three characters: ., - and z. 75 of the 77 characters you wanted are missing, and -, which is outside the range, is added. Index six one-character terms a m z . - / and count the hits:

Other spellings behave the same way (printable ASCII characters accepted, on 10.1.0 against 10.5.2 and main):

• [\!-\~], meaning "printable ASCII without space": 94 on 10.1.0, 3 (! - ~) now.
• [\+-0]: 6 (+ , - . / 0), now 3 (+ - 0).
• [\!--], the range ! to -: 13, now IllegalArgumentException: expected ']' at position 6.
• [\0-\9], which is what RegExp.toString() prints for [0-9]: 10, now 3 (0 - 9).
• Unchanged: [.-z] (unescaped start), [a-\~] (escaped end) and [\-] (an escaped dash on its own).

java.util.regex agrees with 10.1.0 on every row above that it accepts (it rejects \0).

RegExp.parseCharClasses() reads a class one item per loop round (RegExp.java:1312-1348 at ea57360912ed).

• Unescaped item. It reads a character (1326) and then asks "is the next character -?" (match('-'), 1328). If so, it builds a range up to the character after the dash (1333-1334).
• Escaped item. After a backslash (1314), a character that is not a letter or \ goes to a separate branch (1318-1323, comment "escaped character, don't parse it"). That branch adds the character as a single item and never asks about the dash.

So for [\.-z] (positions 0-5 hold [ \ . - z ]):

For [\!--], round 2 reads the first -, sees the second -, and builds a range from - to the next character, which is the closing ]. Nothing is left to close the class, so the parser throws "expected ']' at position 6" (1303).

An escaped range end still works. The end is read by parseCharExp() (1502-1505), which accepts an optional backslash, so [a-\~] is still a range.

parseCharClass() (lines 1203-1212 at releases/lucene/10.1.0) consumed the backslash, read the character with parseCharExp(), and then ran the same match('-') range check for both spellings (1209-1210). #14193 (fe42efc5918, 2025-02-06) rewrote this into the parseCharClasses loop, where an escaped non-letter still reached the range check by falling through to the next loop round. #14227 (19be0e5fd11, 2025-02-13) then added the branch above. Both were first released in 10.2.0.

Which escapes are hit: a backslash followed by any character that is not an ASCII letter and not \. That covers digits, punctuation, space and non-ASCII characters, all executed: [\0-\9] gives 3 characters instead of 10, [\ -z] (escaped space) 3 printable ASCII characters instead of 91, and [\é-ü] only é, - and ü instead of the 20 characters from U+00E9 to U+00FC. An escaped backslash as the start ([\\-z]) was never a range, not even in 10.1.0; that older quirk is separate.

• Who is hit. Any code that passes such a pattern to RegExp:
  • new RegExp(...) itself;
  • RegexpQuery, whose default constructor uses RegExp.ALL (RegexpQuery.java:54-55);
  • the classic QueryParser /.../ syntax: /[\.-z]/ counts 3 instead of 5 on ea57360912ed [exec];
  • not run: the flexible StandardQueryParser, other AutomatonQuery users and downstream engines.
• Default, not opt-in. No flag is involved: the escaped branch is unconditional. [\.-z] misses a with RegExp.ALL and with RegExp.NONE [exec].
• Why people write escaped starts.
  • The javadoc requires escaping reserved characters "also in character classes" (RegExp.java:342-345), so following the docs gives [\.-z].
  • RegExp.toString() itself prints [0-9] as [\0-\9] (on 10.1.0, 10.5.2 and main [exec]). On 10.2.0 and later that string re-parses to 3 characters. Upstream does not promise that toString() output can be re-parsed (rmuir in #14493), so this is only a side effect.
• Silent. No error, and the matched set is wrong in both directions: most of the range is dropped, and a literal - is added. An exception is raised only in a few shapes, such as a range that ends in -.
• Breadth. The candidate's bounded-exhaustive differential tried the 1,867 two-endpoint classes [\s-e] that java.util.regex accepts, with s one of the 31 ASCII punctuation characters other than \, and e >= s a printable ASCII character other than [, ], \ and &. On main and 10.5.2, 1,852 give a different language from java.util.regex, 13 throw, and the 2 that agree do so by coincidence. On 10.1.0: 0 and 0. Space and digit starts are not in that set (java.util.regex rejects \0 to \9); they were run separately and are affected too (see "Which escapes are hit" above).
• Scale. None needed: a six-character pattern triggers it.
• Released. Every release from 10.2.0 to 10.5.2, and main. Moving from 10.1.0 or an older release changes results with no CHANGES or MIGRATE entry. The clean releases that were run are 9.12.2, 9.12.3, 10.0.0 and 10.1.0; by code reading, 7.0.0, 8.0.0 and 9.0.0 have the same range check for an escaped start.
• How common. Unknown. No report was found in the roughly 18 months since 10.2.0. Patterns with an unescaped range start are unaffected, which limits the reach.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected [\.-z] (the range '.'..'z') to accept "a", but it rejects it
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.assertTrue(Assert.java:42)
       org.apache.lucene.util.automaton.TestRegExpEscapedRangeStart.testEscapedRangeStart(TestRegExpEscapedRangeStart.java:28)

The contract, in its own words. These are the RegExp class javadoc lines at ea57360912ed; the same text is in 10.1.0.

• RegExp.java:198-201: charclass ::= charexp - charexp, "(character range, including end-points)".
• RegExp.java:335-336: charexp ::= ... | \ <Unicode character>, "(a single character)".
• RegExp.java:342-345: "The reserved characters used in the (enabled) syntax must be escaped with backslash (\) or double-quotes ("..."). (In contrast to other regexp syntaxes, this is required also in character classes.)"

By this grammar [\.-z] is charexp('\.') - charexp('z'), a range. The escaped form is also the documented way to write a reserved range start.

History.

• #14227 ("Fix failure found by TestOperations.testGetRandomAcceptedString", rmuir, approved by dweiss, merged 2025-02-13) fixed a lenient parse of [\].
• Its commit message says: "When we consume an escape, either it matches the predefined special logic, or we emit range for next(), no other logic: it is an escape."
• Its new tests cover [\-] (a literal dash), [] and [\]. None of them has a - after an escaped character.
• CHANGES.txt has no #14227 entry, and MIGRATE.md says nothing about escapes or ranges in classes.
• In #15182 (2025-09-12) a committer (rmuir) wrote that the syntax "should continue to match https://www.brics.dk/automaton/". The study found that the brics grammar has the same productions.

Tests. No Lucene test builds an escaped range start: TestRegExp.testCoreJavaParity never generates one. With fix option A, every test that was run still passes, including testEscapedDashCharClass: all of util.automaton, search.TestRegexp* and search.TestAutomatonQuery* (266), the queryparser module (533), and 96 tests in 7 other core classes that use RegExp. Tests in other modules that use it (highlighter, monitor, queries, memory, analysis) were not run with the fix.

An independent cross-model review (an independent reviewer from another model family, OpenAI GPT-5.6) was asked to refute the claim. It returned the verdict "real" with high confidence, and it rated the intended-behaviour argument "weak".

The strongest case for "intended":

• The author deliberately wrote "escaped character, don't parse it" and "no other logic: it is an escape".
• testEscapedDashCharClass pins [\-] as a literal.
• RegExp is @lucene.experimental (RegExp.java:352).
• On 10.5.2 and main, [\.-] gives {., -}, as in java.util.regex; 10.1.0 throws on it.
• Just before #14227, escaped range starts worked only by accident: after #14193 (fe42efc5918) rewrote the class parser into a loop, expandPreDefined consumed nothing for a non-letter such as ., so the next loop round re-read it through parseCharExp() and did the range check. #14227 replaced that fall-through, so it may have meant to remove it.

So an escape would make a character an opaque literal, and a dash after it would be a literal too.

Why it fails:

1. "Emit range for next()" describes the one-character item for the escaped character itself. Neither the PR nor its tests mention a following -.
2. The grammar in the same file still defines \ <char> as a charexp and a range as charexp - charexp, and it requires escaping reserved characters in classes. Under the "opaque" reading there would be no documented way to start a range at a reserved character.
3. The parser is not consistent with that reading either. A range end still accepts an escape, so [a-\~] is a range while [\!-\~] is not.
4. No CHANGES or MIGRATE entry announces a change of meaning, and the PR fixed a different input. Being experimental allows API changes, but this silently changed what an existing, documented pattern matches.
5. Fix option A keeps [\-] a literal, and every existing test that was run with it passes (scope under "Tests").
6. The accidental fall-through never reached a release: #14193 and #14227 were merged a week apart and both first released in 10.2.0. Every release up to 10.1.0 had an explicit range check for an escaped start, and #14227's tests and description concern [\], not a range.

A question a maintainer may ask. The grammar also lists \d, \s and \w as a charexp, yet [\d-z] is {0-9, -, z} on both 10.1.0 and main (executed). So charexp - charexp was never applied literally to predefined classes. That does not weaken the claim: the \ <Unicode character> production is explicitly "(a single character)", and it was a range start through 10.1.0.

Verdict: not intended; a regression.

One fix was validated with the gradle build in a scratch worktree at ea57360912ed: option A, the study's idea, restructured. The others are design alternatives and were not run.

In parseCharClasses, an escaped character that is not a letter or \ sets c = next() and then goes through the same match('-') range check, case-insensitive handling and single-character code as an unescaped one. The predefined-class branch ends with continue, which in a do-while goes to the loop condition, as before. The patch is +21/-22 lines in one method, mostly re-indentation.

do {
  int c;
  if (match('\\')) {
    if (peek("\\ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz")) {
      expandPreDefined(starts, ends); // \d \s \w ..., \\, or throws for other letters
      continue;
    }
    c = next(); // escaped character: taken literally, but it can still start a range
  } else {
    c = parseCharExp();
  }
  if (match('-')) { /* range or CASE_INSENSITIVE_RANGE, unchanged */ }
  else if (check(ASCII_CASE_INSENSITIVE | CASE_INSENSITIVE)) { /* case forms, unchanged */ }
  else { /* single character, unchanged */ }
} while (more() && !peek("]"));

Trade-offs

• Restores 10.1.0 behaviour: all 15 rows of the pattern table match 10.1.0 exactly.
• Also fixes registry bug LFV-022 (CASE_INSENSITIVE skipped for an escaped character in a class), because escaped characters now take the shared path: its repro goes from 6 of 7 failing to 7 of 7 passing.
• The same case-folding change applies under the deprecated ASCII_CASE_INSENSITIVE flag: the shared code checks ASCII_CASE_INSENSITIVE | CASE_INSENSITIVE (RegExp.java:1336), and the flag's javadoc says it now behaves like CASE_INSENSITIVE (418-425). It matters only for escaped characters with other case forms, in practice escaped non-ASCII letters such as \é, since escaped ASCII letters go to expandPreDefined. Not run with that flag.
• Some patterns that 10.5.2 and main accept throw again, as in 10.1.0. [\.-] gives {., -} today and throws "expected ']' at position 5" with the fix, as [.-] already does on main. [\~-!] gives {!, -, ~} today and throws "invalid range" with the fix. This deserves a CHANGES line.
• An escaped backslash as the start ([\\-z]) stays a non-range, as in 10.1.0; out of scope.

What the checks validated

• TestRegExpEscapedRangeStart: 6 of 6 pass (stock: 6 tests, 4 failed).
• :lucene:core:test --tests 'org.apache.lucene.util.automaton.*' --tests 'org.apache.lucene.search.TestRegexp*' --tests 'org.apache.lucene.search.TestAutomatonQuery*': 266 tests, 0 failures, 2 skipped. Stock gives 266 tests and 4 failures, all in the repro class.
• :lucene:queryparser:test: 533 tests, 0 failures, 1 skipped (same as stock).
• 7 other core classes that use RegExp or RegexpQuery (TestTermsEnum, TestTermsEnum2, TestDocValuesRewriteMethod, TestFieldCacheRewriteMethod, TestDisjunctionMaxQuery, TestMatchesIterator, TestQueryBuilder), through the repro runner: 96 tests, 0 failures (run by the fact check).
• TestRegExp* and TestRegexp* with seed 1234ABCD: 89 tests pass (the log records the seed only; the command line was not saved).
• TestRegExpParsing with the two new tests: 65 of 65 pass (stock: the 2 new tests fail).
• :lucene:core:checkGoogleJavaFormat passes.
• Through the repro runner (--shadow-src): the repro passes 6 of 6 with -ea and -da, the LFV-022 repro passes 7 of 7, and the pattern table is identical to 10.1.0.
• Not run with the fix: tests in other modules that use RegExp or RegexpQuery (highlighter, monitor, queries, memory, analysis).

Copy the match('-') range and CASE_INSENSITIVE_RANGE handling into the else branch at RegExp.java:1318-1323.

Trade-offs

The smallest behavioural change: escaped single characters still skip case folding, so LFV-022 stays open. It duplicates the range code in two places. Not run.

Amend the javadoc to say that an escaped character cannot start a range.

Trade-offs

It contradicts the grammar and the escaping rule in the same javadoc, and the brics grammar a committer wants to keep. It leaves no documented way to start a range at a reserved character, and it keeps the silent change for everyone who upgraded from 10.1 or earlier.

Method: differential fuzzing (Jazzer) against java.util.regex, the study's non-formal baseline tier, 2026-10-22, main 7cecf79523.

• An AI tool ran Jazzer 0.30.0 on a target, RegexDiffFuzzer, that renders one random regex syntax tree both as a Lucene RegExp and as a java.util.regex pattern and compares the two languages. About 145 k executions in 5 runs produced this bug, LFV-022 and two automaton findings.
• A separate, independent AI verifier read the candidate's repro only for its input, wrote its own tests (6 of 9 fail; the 3 controls pass) and an ASCII differential. It bisected the change at commit level to 19be0e5fd11 (#14227) and corrected the candidate's first guess, #14193, which still parses [\.-z] as a range.
• On 2026-10-25 the hunt re-ran the repro on ea57360912ed (-ea and -da) and on 10.5.2, 10.2.0 and 10.1.0, built the full release matrix and the A/B commit pairs, searched for prior reports, and obtained the cross-model refute review. For this page, the fix was validated with gradle.