JavascriptCompiler: a hex or octal literal above Long.MAX_VALUE, such as 0x8000000000000000, throws NumberFormatException instead of ParseException

Lucene expressions read hex and octal literals as signed 64-bit long values with Long.parseLong, and the grammar accepts them with any number of digits. A literal of 2^63 or more, such as 0x8000000000000000, the 64-bit mask 0xffffffffffffffff or the octal 01000000000000000000000, does not fit: Long.parseLong throws NumberFormatException, nothing in the code generator catches it, and compileExpression passes every runtime exception that is not a wrapped ParseException to the caller unchanged.

So compile throws an unchecked java.lang.NumberFormatException ("For input string: \"8000000000000000\" under radix 16") instead of the ParseException every compile overload documents, and a caller that maps ParseException to "invalid expression" does not handle it. The literal is still rejected: no wrong results, no index effect, no hang. The largest in-range literals, 0x7fffffffffffffff and 0777777777777777777777, compile and evaluate correctly, so the boundary is exactly Long.MAX_VALUE.

The two calls are in the module's first patch (LUCENE-5207, 2013): all 122 lucene-expressions releases from 4.6.0 to 10.6.0 fail, and so does main. Not fixed upstream. Fix: catch the NumberFormatException at the literal and throw ParseException (validated with the module's tests and checks).

1. The grammar accepts any length. Javascript.g4:83 OCTAL: [0] [0-7]+; and :84 HEX: [0] [xX] [0-9a-fA-F]+; have no digit limit, so 0x8000000000000000 is a valid token.
2. The code generator parses with a signed long. visitNumeric (JavascriptCompiler.java:432-445) calls Long.parseLong(ctx.HEX().getText().substring(2), 16) (:434) for hex, Long.parseLong(ctx.OCTAL().getText().substring(1), 8) (:436) for octal and Double.parseDouble (:438) for decimal. Long.parseLong accepts at most 2^63 - 1 and throws NumberFormatException above it; nothing catches it.
3. Source errors normally leave the visitor as a wrapped ParseException. The visitor runs inside the ClassFile.build callback (:255-259), which cannot throw checked exceptions, so for bad source text it throws newWrappedParseException (:857-859), for example for wrong arity (:458) and unknown functions (:505-507).
4. compileExpression unwraps only that case. catch (RuntimeException re) (:260-265) rethrows the cause when it is a ParseException and every other runtime exception unchanged, so the NumberFormatException escapes every public compile overload. It happens before any class is defined (:276), so no Expression exists.

Each failure is a bare NumberFormatException (getCause() is null). Throwing ParseException at the parse site makes the repro pass, so the two unguarded calls are the whole cause.

• What goes wrong. The caller sees an unchecked java.lang.NumberFormatException (a subclass of IllegalArgumentException), "For input string: \"8000000000000000\" under radix 16", with no position, instead of a ParseException that names the expression. A handler that maps ParseException to "invalid expression" (an error shown to whoever typed it) does not run; the exception surfaces as a generic failure or propagates.
• Who is exposed. Any public compile overload (given a valid function map, for the overloads that take one), default settings, assertions on or off, no index needed. In practice a 64-bit mask such as 0xffffffffffffffff or the sign-bit mask 0x8000000000000000 in a bitwise expression. On #3534 (an open request for bitwise filtering on numeric fields), gsmiller suggested expressions for such tests in 2024.
• Partly masked on the overloads with a function map. They document IllegalArgumentException for a bad function signature (JavascriptCompiler.java:169, :191, :215), so a caller that catches IllegalArgumentException there catches this by accident. compile(String) documents no IllegalArgumentException.
• Not affected. Decimal literals, in-range hex and octal literals, results (nothing is computed wrongly) and the index. The recorded security probe found no hang and no residual state for its inputs: a 1 MiB trigger literal costs 9-14 ms, about as much as a valid literal of the same length.
• How likely. Rare: it needs a literal of 2^63 or more, the failure is loud at compile time, and no prior public report was found in the recorded searches, over 13 years of releases.

The test sits in org.apache.lucene.expressions.js next to the compiler's other tests but uses only JavascriptCompiler.compile(String) with the default functions. Both tested seeds, DEADBEEF and 5EED1234, fail the same way (the test uses no randomness), and the same file fails unchanged against the 10.5.2 release jars.

As a regression test: ./gradlew tidy wraps its long comment line (nothing else changes), and catch (ParseException _) is Java 22+ syntax, fine on main; a Java 21 backport needs a named variable. The octal sibling (01000000000000000000000, :436) has the same symptom and fix and is left out on purpose.

TestHexLiteralAboveLongMaxValue > testCompileHexLiteralAboveLongMaxValue FAILED
    java.lang.AssertionError: compile("0x8000000000000000") threw java.lang.NumberFormatException: For input string: "8000000000000000" under radix 16
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:3A1B5B8F06FCBC08]:0)
        at org.apache.lucene.expressions.js.TestHexLiteralAboveLongMaxValue.testCompileHexLiteralAboveLongMaxValue(TestHexLiteralAboveLongMaxValue.java:30)

        Caused by:
        java.lang.NumberFormatException: For input string: "8000000000000000" under radix 16
            at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:67)
            at java.base/java.lang.Long.parseLong(Long.java:594)
            at org.apache.lucene.expressions.js.JavascriptCompiler$2.visitNumeric(JavascriptCompiler.java:434)
            ...
:lucene:expressions:test (FAILURE): 1 test, 1  failure

• The contract. Every compile overload says @throws ParseException on failure to compile (JavascriptCompiler.java:150, :168, :190, :214). IllegalArgumentException is documented only for a bad function signature, IllegalStateException only for link failures. The class javadoc sends the one documented parser limit, nesting, to ParseException (:104-106).
• The literal is legal. Javascript.g4:83-84 has no digit limit, and package-info.java:25 lists "Integer, floating point, hex and octal literals" with no range. No javadoc, CHANGES or MIGRATE entry states a range or mentions NumberFormatException.
• Tests. testInvalidCompiles (TestJavascriptCompiler.java:141-172) requires ParseException for 0y1234, 500EE and 500.5EE. No test uses a literal of 2^63 or more.
• Precedent. LUCENE-6965 (#8022, 5.5.0) moved bad function names and arity from IllegalArgumentException to ParseException; its description says that "would be better since that's the exception clients will be prepared to receive". #16423 (10.6.0, Uwe Schindler and Seth Kraft) replaced StackOverflowError with ParseException and clarified the documented exceptions, without this case. No commit or discussion mentions literal range.

The strongest case for "intended" (built by the devil's-advocate verifier and both executed cross-model reviews): JavascriptCompiler is @lucene.experimental (:111), the syntax is only "based on" JavaScript, hex and octal literals have been signed 64-bit values since 2013, NumberFormatException is an IllegalArgumentException, the usual Java signal for a bad argument, and @throws lists for unchecked exceptions are often incomplete.

Why it fails: that justifies rejecting such literals, not the exception type. The bad input is expression source text, the class of error LUCENE-6965 moved to ParseException; the compiler already has a helper for it and uses it for other source errors; compile(String) documents no IllegalArgumentException; and the exception list was re-stated in 2026 without this case. Verdict: a bug in the exception type. The independent reproduction, the intended-behaviour check and both executed cross-model reviews rated the intended argument weak; the adjudication gave no rating but found that it "justifies rejecting such literals but not the exception type". A later fact check by another model family (OpenAI GPT-5.6) rated it moderate: keeping hex and octal literals in the signed long range is a plausible, long-standing design choice, and a grammar that accepts any length is not a promise that any magnitude is supported. So the evidence argues only for the exception type: it is inconsistent with the documented failure channel and with LUCENE-6965, but it does not show what the original authors intended for out-of-range literals.

One option was built and validated through gradle on 645d4fe14a20; a second is a behaviour change left to maintainers; a third is not a fix.

A parseIntegerLiteral helper in the visitor replaces the two Long.parseLong calls in visitNumeric (+20/-2 in JavascriptCompiler.java). It throws ParseException: Invalid expression '0x8000000000000000': integer literal out of range (0x8000000000000000) with error offset 0 (4 for 1 + 01000000000000000000000), wrapped in a RuntimeException that the existing code at :260-265 unwraps. Chaining the cause follows the StackOverflowError catch at :266-271. The snippet is condensed.

  // visitNumeric, for HEX (OCTAL the same with substring(1) and radix 8):
- pushLong(Long.parseLong(ctx.HEX().getText().substring(2), 16));
+ pushLong(parseIntegerLiteral(ctx, ctx.HEX().getText().substring(2), 16));

  private long parseIntegerLiteral(NumericContext ctx, String digits, int radix) {
    try {
      return Long.parseLong(digits, radix);
    } catch (NumberFormatException e) {
      var pe = new ParseException("Invalid expression '" + sourceText
          + "': integer literal out of range (" + ctx.getText() + ")", ctx.start.getStartIndex());
      pe.initCause(e);
      throw new RuntimeException(pe);
    }
  }

Trade-offs

• Only the failing case changes. Every literal that compiles today parses exactly as before.
• The message embeds the whole source like the other Invalid expression '...' messages (:458, :505-507), and then the literal: a 1 MiB literal gives a message of about 2 MiB (measured with the study's sketch, whose message is the same). Quoting only the literal would keep it short.
• It catches NumberFormatException at the parse site only, not IllegalArgumentException in general, in line with a committer's remark on LUCENE-6965 that something unexpected inside the visitor could throw an IllegalArgumentException.
• The study's shorter sketch (a catch (NumberFormatException nfe) that ignores nfe) fixes the repro too, but fails Lucene's ecjLintMain: "The value of the exception parameter nfe is not used". Chaining the cause, as here, avoids that.

What the checks validated

• Gradle, scratch worktree of 645d4fe14a20 (one checkout, OpenJDK 25.0.3, Linux aarch64), seed DEADBEEF: the repro fails on stock and passes with fix A, assertions on and off.
• The whole :lucene:expressions:test with fix A and the repro (assertions on): 14 suites, 171 tests, 0 failures, 0 skipped (the 13 upstream suites with 170 tests, including TestJavascriptCompiler, plus the repro).
• The closest suite TestJavascriptCompiler and the repro with -Ptests.asserts=false: 21 tests pass.
• :lucene:expressions:check -x test (ANTLR checksum, google-java-format, ecj lint, forbidden APIs): passes, with the repro's comment wrapped by tidy.
• A probe on the standalone runner shows the new ParseException, its offset and the chained cause, for the hex and the octal trigger.

Parse the digits with new BigInteger(digits, radix) and, when the value does not fit a long, load BigInteger.doubleValue(), the way JavaScript reads 0x8000000000000000 as 2^63.

Trade-offs

A behaviour change (today's refusals start to compile) that needs a CHANGES entry and a rule for bitwise positions: there the operand becomes a long again, and the JVM's double-to-long conversion saturates at Long.MAX_VALUE, so x & 0xffffffffffffffff would silently mean x & 0x7fffffffffffffff unless such literals are rejected there. Not built or run.

Read the 64 bits as a two's-complement long, as Java's 0xffffffffffffffffL does.

Trade-offs

0xffffffffffffffff would become -1: 1 + 0xffffffffffffffff would be 0, and a filter such as x > 0xffffffffffffffff would turn today's refusal into a silent match of every non-negative value.

Method: semi-formal (a hunt unit, the expressions compiler).

• The finder, an AI tool, was writing the literal semantics of an exact reference model of JavascriptCompiler, used to check every small expression tree against the compiled code, and noticed that hex and octal tokens go through signed Long.parseLong with no handler.
• No formal or exhaustive tool output produced the signal: the model's literal alphabets stay inside the long range (the largest is 0x7fffffffffffffff), and the unit's grammar fuzzer skips this NumberFormatException on purpose as already known, so it neither confirms nor adds to it.
• The finder then confirmed it with a probe and a public-API JUnit on the stock jar; independent reproducers wrote their own tests from the claim alone, and the one-test repro was cut from those on 2026-10-29.