JavascriptCompiler.compile(source, functions, Integer.MAX_VALUE) rejects every expression, even "1": maxNestingDepth + 1 overflows

#16423 added a public overload JavascriptCompiler.compile(String, Map<String, MethodHandle>, int maxNestingDepth) (JavascriptCompiler.java:195-199 on main 645d4fe14a20), documented only as "the maximum depth of nesting (function calls, precedence - also implicit)" (:187-188). Its parse-time guard is installed as new JavascriptNestingDepthListener(sourceText, maxNestingDepth + 1) (:319-320), because the listener "will count root node" (:317-318). For Integer.MAX_VALUE the int addition wraps to Integer.MIN_VALUE, and the listener's depth > maxNestingDepth (JavascriptNestingDepthListener.java:44-45) is true on the first rule the parser enters, the root: every expression, even 1, fails with ParseException: Invalid expression '1': Nesting level too deep (>-2147483648). Integer.MAX_VALUE - 1 works, and the one- and two-argument overloads (limit 1024) cannot be hit. The failure is loud and the same with and without assertions; no value is ever wrong. The overload is on main, branch_11x, branch_10x and branch_10_6 (10.6.0, released 2026-10-29, fails too); no release through 10.5.2 has it. Not passing the extra level when the limit is already Integer.MAX_VALUE fixes it.

• maxNestingDepth: the limit argument of the public overload compile(String, Map, int) (JavascriptCompiler.java:195-199), "the maximum depth of nesting (function calls, precedence - also implicit)" (:187-188). The one- and two-argument overloads pass DEFAULT_MAX_NESTING_DEPTH = 1024 (:138, :154-176). The value is stored as given (:219-240).
• Parse-time guard: a JavascriptNestingDepthListener attached to the ANTLR parser (:319-320). enterEveryRule does depth++ and throws when depth > maxNestingDepth (JavascriptNestingDepthListener.java:43-53); exitEveryRule does depth--.
• Visitor-side guard: a second listener instance with the raw maxNestingDepth (:404), driven by the code generator as it walks the tree.

1. The + 1. getAntlrParseTree creates the parse-time guard with maxNestingDepth + 1; the comment says it "will count root node, so add one recursion extra" (:317-320).
2. The wrap. For maxNestingDepth = Integer.MAX_VALUE = 2,147,483,647 the sum is 2,147,483,648, which does not fit an int and wraps to Integer.MIN_VALUE = -2,147,483,648.
3. The first rule fails. javascriptParser.compile() (:321) enters the root rule first (JavascriptParser.java:217). depth becomes 1 and 1 > -2147483648 is true, so the listener throws before the text of the expression matters. The message prints the wrapped limit: "Nesting level too deep (>-2147483648)".
4. Only this value. Every other int plus 1 is representable. Limits 0 and below reject every expression, which matches a maximum depth of 0 or less. The visitor-side guard (:404) has no + 1 and is never reached here.

• compile("1", DEFAULT_FUNCTIONS, Integer.MAX_VALUE - 1).evaluate(null) returns 1.0; with Integer.MAX_VALUE: ParseException: Invalid expression '1': Nesting level too deep (>-2147483648), with assertions on and off.
• The same for 1 + 2 (3.0 at Integer.MAX_VALUE - 1), sqrt(16), 1 ? 2 : 3 and a 1,100-deep nest of parentheses. By the code, any source that reaches the parser's root rule is rejected there, whatever its text.
• 1 compiles at every sampled limit from 1 to 2,147,483,646 (1, 2, 3, 4, 8, 16, 1024, 2048, 1,048,576, 2,147,483,645, 2,147,483,646) and, of the positive limits, fails only at 2,147,483,647.
• A 1,500-deep expression fails at the default 1024, compiles at 4,096 and at Integer.MAX_VALUE - 1 (on a 256 MB stack) and fails at Integer.MAX_VALUE.

• Who is exposed. Callers of compile(String, Map, int) who pass Integer.MAX_VALUE, a natural value for "no practical limit" (Lucene's own JoinUtil documents it as the value that disables its bound), for example after the default 1024 rejected a deep generated expression. The PR author made the limit configurable "so people can tune it". The package-private compile(String, Map, int, boolean) (JavascriptCompiler.java:219-227) takes the same path but is reachable only from inside the package; in the tree only CompilerTestCase calls it, with the default limit.
• Opt-in. The one- and two-argument overloads use 1024 (listener limit 1025) and cannot hit it. No index, codec or data is involved; single thread, deterministic.
• Effect. The expression feature is unusable for that caller until the value changes; it shows on the first compile. No wrong value, no hang, no lasting state.
• Raising the limit is meaningful. A 1,500-deep expression that the default rejects compiles at 4,096 and at Integer.MAX_VALUE - 1. With Integer.MAX_VALUE - 1 and very deep input (a 20,000-deep nest on the default stack) the existing StackOverflowError fallback gives a clean ParseException.
• How likely. Plausible but narrow: a natural value, a new API, no known caller yet. Workaround: pass Integer.MAX_VALUE - 1 or any large finite limit.
• Timing. 10.6.0 RC1 contains it; the vote passed and 10.6.0 was released from it on 2026-10-29, so a fix can come in a 10.6.x release or 11.0.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 4 test(s) run, 1 failed
    => java.lang.AssertionError: expected "1" to compile with maxNestingDepth=Integer.MAX_VALUE, but got: java.text.ParseException: Invalid expression '1': Nesting level too deep (>-2147483648)
       __randomizedtesting.SeedChain.seed([DEADBEEF:90DC4BAD2263C875:A1AD4638E010A3D3])
       org.apache.lucene.expressions.js.TestMaxNestingDepthMaxValue.testMaxValueLimitCompilesTrivialExpression(TestMaxNestingDepthMaxValue.java:31)
     Caused by: java.text.ParseException: Invalid expression '1': Nesting level too deep (>-2147483648)
       org.apache.lucene.expressions.js.JavascriptCompiler.newWrappedParseException(JavascriptCompiler.java:858)
       org.apache.lucene.expressions.js.JavascriptNestingDepthListener.enterEveryRule(JavascriptNestingDepthListener.java:46)
       org.antlr.v4.runtime.Parser.triggerEnterRuleEvent(Parser.java:396)
       org.antlr.v4.runtime.Parser.enterRule(Parser.java:630)
       org.apache.lucene.expressions.js.JavascriptParser.compile(JavascriptParser.java:217)
       [...]

• Contract. @param maxNestingDepth "the maximum depth of nesting (function calls, precedence - also implicit)" (JavascriptCompiler.java:187-188); no range, no reserved value, and nothing validates it (:195-199, :219-240). The class javadoc only says the default of 1024 is enforced with a ParseException (:104-109). CHANGES (10.6.0, lucene/CHANGES.txt:551-555): "The new code allows to add a hard limit for nesting (default is 1024) and no longer throws virtual machine errors."
• Monotonicity. A larger limit cannot rightly accept less: Integer.MAX_VALUE - 1 accepts 1, Integer.MAX_VALUE accepts nothing.
• The + 1 is root-node compensation. The comment says so (:317-318), and the visitor-side guard uses the raw value (:404).
• Too-high limits are expected. The StackOverflowError fallback (:266-272) carries the comment "we should catch this before in the visitor, but too high limits may cause this." The #16423 body calls it a "last safety"; in review uschindler wrote "I am not sure if 250 is enough for all cases, so I'd like to have it configurable (at least for a while, so people can tune it)". In #16549 (dweiss, test-only) the quoted fix note says "the SOE fallback in production code is intentional and unchanged".
• Precedent. In review msokolov compared the limit with "BooleanQuery's 1024 limit"; that limit, now IndexSearcher.setMaxClauseCount, accepts any value >= 1, Integer.MAX_VALUE included (IndexSearcher.java:256-261). JoinUtil.createJoinQuery documents that min 0 and max Integer.MAX_VALUE "disables the min and max 'from' documents filtering" (JoinUtil.java:512-517).
• History and tests. The overload, the listener and the + 1 arrived together in b6c2c31e225 (#16423); its body, 3 comments, 9 reviews and 7 inline comments never mention Integer.MAX_VALUE, the + 1 or argument checks. No upstream test passes a non-default limit (CompilerTestCase.java:37-38 and the testRecursionDepth* tests use 1024).

The strongest case for "works as designed" (built by the intended-behaviour checker and the reviews by another model family): the class is @lucene.experimental (JavascriptCompiler.java:111); the limit is a resource guard; Integer.MAX_VALUE levels can never be reached before the JVM stack runs out (uschindler in review: "In my testing 2048 sometimes causes stack overflows already"), so a maintainer could say "pass a large finite number", or prefer to reject the value.

Why it fails: that argues how useful the value is, not what it should do. Nothing documents or rejects it, Integer.MAX_VALUE - 1 (just as unreachable) works, and the code keeps an intentional StackOverflowError-to-ParseException fallback for limits too high to stop the recursion first (:266-272). What remains is severity (one value, a loud failure), which the Low priority reflects. Nothing documents Integer.MAX_VALUE as a "no limit" value either. Verdict: a defect; rejecting the value with a clear IllegalArgumentException resolves it as well as accepting it.

One fix was built and validated with gradle; the alternatives were not built. Rejecting the value instead (option B) is an equally valid answer for maintainers.

JavascriptCompiler.java:317-320, one file, +4/-2 with the comment (after tidy):

    // add listener to detect too deep nesting early, this may not catch all occurrences... (this
    // will count root node, so add one recursion extra, without overflowing Integer.MAX_VALUE).
    javascriptParser.addParseListener(
        new JavascriptNestingDepthListener(
            sourceText,
            maxNestingDepth == Integer.MAX_VALUE ? maxNestingDepth : maxNestingDepth + 1));

Trade-offs

• Nothing changes for any other limit.
• At Integer.MAX_VALUE the call then behaves like Integer.MAX_VALUE - 1 today: the parse-time guard can never fire (no int depth exceeds Integer.MAX_VALUE), and very deep input is stopped by the existing StackOverflowError fallback with a ParseException (with the same expression shadowed in: a 200,000-deep nest in 6.8 ms and a 1,000,001-character chain in 167 ms, each a ParseException caused by a StackOverflowError). That is the cost of the huge limit the caller chose.
• Not covered: limits of 0 and below, which reject every expression today; a lower-bound check (option B) is complementary.

What the checks validated

• Scratch worktree of main 645d4fe14a20, gradle: the new test fails on the stock tree (the only failure among the module's 174 tests) and passes with the fix, with assertions on and off. The whole lucene/expressions module passes with assertions on and off: 14 suites (13 upstream plus the new test), 174 tests (170 upstream), 0 failures; TestJavascriptCompiler 20 of 20. :lucene:expressions:check -x test (google-java-format, ecjLint, forbiddenApis) passes.
• Standalone runner with the fixed class shadowed in: the new test passes. Earlier, with the finder's identical expression: the finder's test, the independent reproducer's 5 tests and the cross-model reviewers' tests pass, and the 170 upstream expressions tests pass.

Document a range and throw IllegalArgumentException outside it, for example for maxNestingDepth < 1 || maxNestingDepth == Integer.MAX_VALUE.

Trade-offs

Loud at the call site and explicit, but it rejects a "no limit" value that comparable Lucene limits accept (JoinUtil, IndexSearcher.setMaxClauseCount), and it changes today's behaviour for 0 and negative limits (they reject every expression now). Can be combined with the first option for the lower bound. Not built.

Make JavascriptNestingDepthListener's limit a long and pass maxNestingDepth + 1L (two files).

Trade-offs

Same effect as the first option; the message would then print 2147483648. Not built.

Let the parse-time listener skip the root rule, so both guards take the caller's limit unchanged.

Trade-offs

More code. It would also change a side note: today the parse-time message prints the internal limit, for example "Nesting level too deep (>1025)" for the default 1024 (not claimed as a separate bug). Not built.

Method: non-formal (the hunt's expressions-compiler hunt unit, 2026-10-26, main ea57360912ed). The unit's boundary harness checks depths around the default limit (1022 to 1026 and beyond) and could not reach this value; the signal came from reading getAntlrParseTree while writing it. The finder's JUnit test (1 + 2, with the Integer.MAX_VALUE - 1 control) failed with assertions on and off, a main() form failed too, and a one-line shadow fix made it pass with the 170 upstream expressions tests.