Polygon.fromGeoJSON throws StackOverflowError instead of ParseException on deeply nested JSON

Polygon.fromGeoJSON(String) declares only ParseException, but its parser, SimpleGeoJSONPolygonParser, reads each nested JSON array or object with one more recursive call (parseArray and parseObject call each other) and has no depth limit and no Error handler. In a fresh JVM on the default 2040 KiB thread stack (Linux aarch64, OpenJDK 25.0.3), a valid Feature whose properties hold an array nested 10,000 deep (20,146 chars) throws java.lang.StackOverflowError; 5,000 levels return the polygon, and with a 1 MiB stack 5,000 levels fail. Nested object properties and malformed unterminated arrays fail the same way. No wrong polygon is returned and no index is touched; the defect is the undeclared Error, which passes catch (ParseException) and catch (Exception), from an API that gives the caller no limit to set. It has been in every release since 6.2.0, when the parser was added. A depth limit that throws ParseException, as #16423 added to the expressions parser, fixes it.

1. Polygon.fromGeoJSON returns new SimpleGeoJSONPolygonParser(geojson).parse() (Polygon.java:255-257), and parse() starts with parseObject("") (SimpleGeoJSONPolygonParser.java:63-65).
2. parseObject reads each member; a value starting with [ goes to parseArray(newPath) (:138-145), one starting with { to parseObject(newPath) (:146-154). That includes members under properties, which the parser reads only to skip them (it acts only on type, coordinates and crs, :125-128, :174-215).
3. parseArray sends a nested [ to parseArray(path + ".[]") (:308-309) and a nested { to parseObject(path + ".[]") (:310-313).
4. Neither method has a depth parameter, field or check, and the class's only catch is for NumberFormatException in parseNumber (:343-348). Each level holds a frame until its closing bracket; when the stack is full the JVM throws StackOverflowError, and it leaves fromGeoJSON, which declares only ParseException (Polygon.java:255).
5. A smaller second path: for nested arrays under coordinates, the parse can finish and parsePoints then builds an error message with the nested list (:254-263), whose toString() recurses the same depth (seen at 2,000 levels on a 512 KiB stack only). A depth limit bounds this too (by reasoning; the fix was not run on this shape).

A static scan of the stock bytecode (JDK ClassFile API) finds exactly one recursive cycle in the class, {parseArray, parseObject}, with no depth counter and no Error handler.

Fresh JVM per line, -da, main thread, stock 645d4fe14a20 core jar, OpenJDK 25.0.3 on Linux aarch64; the input is the repro's Feature with properties.p nested depth arrays deep:

These are sample depths, not the overflow point. An independent fact check (3 fresh JVMs per depth, same Feature, main thread) puts it between 6,500 levels (13,146 chars, polygon 3 of 3) and 7,000 (14,146 chars, StackOverflowError 3 of 3, also with -ea) on the default stack, and between 3,500 and 4,000 levels (8,146 chars) on 1 MiB.

Other shapes on the default stack (independent reproducer, -da; the hunt's 420-point sweep, -ea and -da): a nested object property {"a":{"a":...}} is fine at 5,000 and fails at 10,000 (60,141 chars); unterminated {"type":"Polygon","coordinates":[[[[... gives ParseException at 5,000 and the Error at 10,000. 1,000 levels never failed. The exact depth varies with stack size, the thread, JIT state and version: the diagram's runs, parsing on a new thread with the same default stack, saw 10,000 levels overflow in 9 of 10 runs, and 6.2.0 needs 20,000 levels on the default stack for the valid shapes; x86 was not run.

• Who. Code that turns request-supplied GeoJSON into Polygons through fromGeoJSON, which the Polygon class javadoc recommends for GeoJSON text (Polygon.java:23-26). Nothing in Lucene calls it outside tests. An independent verifier read one open-source search server that calls it on document field values and catches only ParseException (read, not run).
• Default vs opt-in. No setting is involved; it is the only behaviour of a public static API. Polygon is @lucene.experimental (Polygon.java:41).
• Scale. On the main thread of a fresh JVM the array property overflows from about 7,000 levels (14 KB) on the default 2040 KiB aarch64 stack and from about 4,000 (8 KB) on a 1 MiB stack (3 of 3 JVMs at each depth); 10,000 and 5,000 are the round depths first measured. The exact depth varies with the stack, the thread, JIT state and version.
• No way to bound it through Lucene. fromGeoJSON has no depth or size parameter. A length cap is a poor substitute: at about 22 chars per vertex ([100.123456,0.123456],) a 1,000-vertex polygon is already about 22 KB, more than the 14 KB that already overflowed. Workarounds include pre-parsing with another JSON parser or counting the bracket depth before the call, a thread with a larger stack, or catching StackOverflowError, which maintainers discourage (Robert Muir and Mike Sokolov on #752).
• Effect. An Error in the calling thread only: no wrong polygon, no index change, no shared state (the parser is a new object per call). What follows depends on how the caller handles Error.
• Likelihood: contrived. A MultiPolygon's coordinates are 4 arrays deep, inside a Feature inside a FeatureCollection; nobody writes thousands of levels. The documented searches found no report to Lucene in ten years.

LuceneTestCase is only the base class; the input is fixed (the randomized runner only picks environment settings). The package is org.apache.lucene.geo only so that the file sits next to TestPolygon; Polygon.fromGeoJSON is public. 50,000 levels is well above the fresh-JVM overflow point (about 7,000 levels on the default 2040 KiB stack) and also overflows after JIT warm-up (10 of 10 in one JVM, with -ea and -da). The test runs on the forked test JVM's default thread stack (Lucene's build sets no -Xss), so a test JVM configured with a much larger stack could let the unfixed parser pass. It is not deeper on purpose: on a JVM with a larger stack, the parser's per-level path strings would make heap use grow with the square of the depth reached.

Regression test: this same method, in TestPolygon or as its own class; with the recommended fix the parse ends in ParseException and the test passes.

cp $REPRO/TestGeoJSONDeepNesting.java $LUCENE/lucene/core/src/test/org/apache/lucene/geo/
cd $LUCENE && ./gradlew :lucene:core:test --tests TestGeoJSONDeepNesting -Ptests.seed=DEADBEEF -Ptests.asserts=true
# assertions off fails the same way: -Ptests.asserts=false

# gradle, stock 645d4fe14a20, -Ptests.seed=DEADBEEF -Ptests.asserts=false (stack trimmed; -Ptests.asserts=true fails the same way)
java.lang.AssertionError: expected the polygon or ParseException, got StackOverflowError
	at __randomizedtesting.SeedInfo.seed([DEADBEEF:DDD1C89C8A6B3F85]:0)
	at org.apache.lucene.geo.TestGeoJSONDeepNesting.testFeatureWithDeeplyNestedArrayProperty(TestGeoJSONDeepNesting.java:37)
Caused by: java.lang.StackOverflowError
	at org.apache.lucene.geo.SimpleGeoJSONPolygonParser.parseArray(SimpleGeoJSONPolygonParser.java:285)
	at org.apache.lucene.geo.SimpleGeoJSONPolygonParser.parseArray(SimpleGeoJSONPolygonParser.java:309)
	at org.apache.lucene.geo.SimpleGeoJSONPolygonParser.parseArray(SimpleGeoJSONPolygonParser.java:309)
	... (1,023 recorded parseArray:309 frames)

:lucene:core:test (FAILURE): 1 test, 1  failure

# fresh-JVM probe on the stock jar, default 2040 KiB stack, -da
depth=5000 chars=10146 OK polygons=1
depth=10000 chars=20146 StackOverflowError

• Declared outcomes. Polygon.java:248-257: "Parses a standard GeoJSON polygon string. The type of the incoming GeoJSON object must be a Polygon or MultiPolygon, optionally embedded under a "type: Feature"", and the method declares only ParseException.
• Nested properties are accepted on purpose. The class comment shows a Feature with "prop1": {"this": "that"} in properties (SimpleGeoJSONPolygonParser.java:23-45); TestPolygon.testGeoJSONPolygonFeature parses that Feature, and testPolygonPropertiesCanBeStringArrays (LUCENE-8964, #10007) an array property. RFC 7946 section 3.2 lets properties be any JSON object.
• Depth alone decides. The same unterminated text gives the parser's own "hit EOF" ParseException at 5,000 levels and the Error at 10,000.
• Precedent. lucene/CHANGES.txt:551-555 (10.6.0): GITHUB#16423 (Uwe Schindler, Seth Kraft) added a nesting limit (default 1024) to the expressions parser, which "could throw StackOverflowError on deeply nested expressions" and now "no longer throws virtual machine errors". RFC 8259 section 9 allows a parser to limit nesting depth; a limit that throws the declared ParseException is all this asks.
• Nothing makes the Error an outcome. No test covers nesting depth (lucene/core/src/test/org/apache/lucene/geo), and the introducing issue #8433 never discusses it.

The strongest case for "intended": SECURITY.md line 38 makes limits on "deeply nested queries" and similar untrusted input "the responsibility of the downstream application"; the parser "Does minimal parsing" (SimpleGeoJSONPolygonParser.java:47-51) and already rejects some valid JSON (true, false or null inside an array, :318-321); Polygon is @lucene.experimental; a throws clause never rules out a VirtualMachineError; LUCENE-6156 (#7218), a RegExp StackOverflowError, was resolved "Not A Problem"; and #16423 was labelled an enhancement.

Why it fails: line 38 rests on "Lucene provides mechanisms to bound this", and fromGeoJSON has none; line 46 adds that such issues "are still valid bug reports and hardening suggestions". LUCENE-6156 was about a stack a downstream product had shrunk (as Robert Muir noted there); this fails on the default stack. The request is a ParseException at a depth limit, not that any depth must parse, so "minimal parser" does not explain the Error. Verdict: a real, low-priority robustness bug, of the kind #16423 fixed as an enhancement.

Option A was written and validated in a scratch worktree of 645d4fe14a20; option B is the hunt's sketch, validated earlier with the parser shadowed. The limit value is the maintainers' choice.

SimpleGeoJSONPolygonParser.java only. parse() calls parseObject("", 1); parseObject and parseArray take an int depth, call checkDepth(depth) first and pass depth + 1 at their four recursive calls. The depth counts every array and object from the outer object, and the check runs before the bracket is consumed, so the exception offset points at the first bracket past the limit.

  /** Maximum nesting depth of arrays and objects; deeper input throws {@code ParseException}. */
  static final int MAX_NESTING_DEPTH = 1000;

  /** Throws {@code ParseException} instead of recursing deeper than {@link #MAX_NESTING_DEPTH}. */
  private void checkDepth(int depth) throws ParseException {
    if (depth > MAX_NESTING_DEPTH) {
      throw newParseException("maximum nesting depth of " + MAX_NESTING_DEPTH + " exceeded");
    }
  }

  // parseObject(String path, int depth) / parseArray(String path, int depth) start with checkDepth(depth)
  // and recurse with depth + 1; parse() starts with parseObject("", 1).

Trade-offs

• Smallest change with the same frames per level as today; the cost is one int compare per array or object.
• 1,000 is far above any ordinary GeoJSON structure; it does deliberately reject exceptionally deep but valid JSON, which RFC 8259 section 9 allows. The value is a judgement call: #16423 uses 1024 (configurable), and jackson-core's default limit is 1000.
• It does not address the separate heap growth from long keys above about 900 levels (LFV-952), which needs its own fix.

What the checks validated

Gradle --no-daemon in a scratch worktree of 645d4fe14a20, OpenJDK 25.0.3 (2026-10-31). Stock: the repro fails with assertions on and off. With the fix:

• the repro passes with assertions on and off;
• org.apache.lucene.geo.* (18 classes, including the repro, TestPolygon 16, TestTessellator 77 with its GeoJSON test polygons, TestSimpleWKTShapeParsing 14): 194 tests, 0 failures, 7 skipped, with seeds DEADBEEF and CAFEBABE;
• checkGoogleJavaFormat, forbiddenApisMain and ecjLintMain of lucene/core pass;
• a fresh-JVM probe on the compiled fix: 998 arrays inside properties.p (1,000 levels in all) return the polygon; 999, 10,000 and 50,000 give ParseException: maximum nesting depth of 1000 exceeded at character offset 1142 (the bracket that opens level 1,001); the same on a 512 KiB stack and with -ea.

The whole lucene/core test suite was not run.

A depth field, an enter() that throws past MAX_NESTING_DEPTH = 1000, and parseObject / parseArray each split into a wrapper with try { ... } finally { depth--; } and a body. Same behaviour as the recommended option.

Trade-offs

Two methods per level (a wrapper frame each) and a larger diff for the same effect; the finally is not needed, because any ParseException ends the whole parse.

What the checks validated

With the parser shadowed in a standalone runner on main: the hunt's 3-shape JUnit passes 3 of 3, upstream TestPolygon 16 of 16, and the minimal repro passes with -ea and -da on 645d4fe14a20.

An overload such as Polygon.fromGeoJSON(String, int maxNestingDepth), as #16423 made its limit configurable.

Trade-offs

Adds public API to an experimental class for a limit ordinary GeoJSON never reaches.

An explicit stack instead of recursion.

Trade-offs

Removes the stack limit but rewrites the parser; deep input then costs only heap (quadratic with today's per-level path strings). No benefit for real GeoJSON.

Wrap the parse in catch (StackOverflowError e).

Trade-offs

Maintainers argued against catching the Error for RegExp on #752 and preferred detecting the problem before it happens; it also unwinds a full stack on every bad input.

Method: semi-formal.

• Lead. A probe for text parsers that might throw virtual machine errors on deep nesting, the class #16423 fixed in lucene/expressions, ran the stock jars and saw StackOverflowError at 5,000-20,000 GeoJSON levels.
• Static check. A recursion-guard scan of the stock bytecode with the JDK ClassFile API: one recursive cycle {parseArray, parseObject}, no depth counter, no Error handler, prediction "unguarded recursion".
• Model-based sweep. 420 points, one fresh JVM each: 10 input families (6 GeoJSON, 4 WKT; the WKT polygon-parentheses control never mismatched), depths 1,000 to 100,000, 512 KiB, 1 MiB and default stacks, -ea and -da, each compared with a depth-independent expected outcome (a result for valid input, ParseException for malformed input).
• Reduced to a public-API JUnit, then to the one-test minimal repro.