Lucene's GeoJSON parser remembers where it is with a text address, and makes a new, longer copy of it at every nesting level.
- You call
Polygon.fromGeoJSONon a valid GeoJSON Feature: a 4-point Polygon geometry, andpropertieswith one member whose key is 750,000 chars long and whose value is 900 nested empty arrays[[[...]]]. The text is 751,925 chars. - You expect the one polygon back:
propertiesdo not change the polygon (the parser only walks through them). With a 2 GiB heap, or with the same 900 levels under a 1-char key, that is what you get. - At Lucene's default 512 MiB test heap you get
OutOfMemoryError: Java heap space, with assertions on and off. - Why: to know where it is, the parser carries a text address such as
properties.kkk….[].[]and makes a new, longer copy at every level, keeping each copy until that level is done, so all 900 are alive at once. 900 copies of a 750,000-char address are 676 million chars, about 645 MiB. Yet the parser only compares the address with five short constants and acrsprefix.
Only applications that pass GeoJSON they do not control to Polygon.fromGeoJSON, an experimental API that no other Lucene code calls. Shallow GeoJSON with short keys is not at risk. But the usual input limits do not protect: the repro is 752 KB and 902 levels deep, and 1 MB of 1,000 nested objects with 1,000-char keys also exhausts 512 MiB.
Mechanism, terms and numbers595 words
SimpleGeoJSONPolygonParser, behind Polygon.fromGeoJSON, tracks its position with a path String (path + "." + key for an object member, path + ".[]" for an array element), and every recursive frame keeps its own copy alive until it returns, so live memory is about nesting depth times path length. A valid Feature whose properties hold one 750,000-char key over 900 nested empty arrays (751,925 chars, 902 levels) needs about 676 million chars of path strings at once and fails with OutOfMemoryError: Java heap space at Lucene's default 512 MiB test heap, with assertions on and off; it parses at 2 GiB, and the same nesting under a short key, or the same text as a string value, parses at 512 MiB. The path is only ever compared with five short constants and a crs prefix, so keeping its first 21 chars gives identical behaviour with linear memory. Present since the API was added in 6.2.0; ordinary, shallow GeoJSON with short keys is not at risk.
parseObject(String path)(SimpleGeoJSONPolygonParser.java:100; comment at:99: "path is the \"address\" by keys of where we are, e.g. geometry.coordinates").parse()starts with""(:65).- For a member whose value is an array or object it builds a new String,
newPath = path + "." + key(:143,:151; justkeyat the top,:141,:149), and recurses with it (parseArray(newPath)at:145,parseObject(newPath)at:153). parseArray(String path)(:283) passespath + ".[]"to every nested array (:309) or object (:312).- Each frame still needs its
pathwhile its child runs:parseArrayuses it again for the next element (inside the loop),parseObjectafter the child returns (:174,:186,:205). So at depth d all d Strings are live at once.
path.equals("crs.properties") (:125, :174), path.startsWith("crs") (:186), path.equals("features.[]") || path.isEmpty() (:197), isValidGeometryPath (:192, :194, :205) = isEmpty() || equals("geometry") || equals("features.[].geometry") (:222-224), and path.length() == 0 (:140, :148). The longest constant is 20 chars. Error messages quote the input, not the path.
In the repro the first array's path is properties. + key = 750,011 chars, and each deeper level adds .[]. 900 such Strings total 900 x 750,011 + 3 x 899 x 900 / 2 = 676,223,550 chars; the JDK stores these Latin-1 Strings at one byte per char, so about 645 MiB, roughly 900 times the input. In general: about d x K + 1.5 d^2 bytes for d nested arrays under a K-char key, and about K x d^2 / 2 for d nested objects each under a K-char key. The hunt's model predicted the heap threshold of three shapes with the same d x K product (250 x 1 MiB, 500 x 512 KiB, 1,000 x 256 KiB) at 252-253 MiB; bisection on the stock jar measured 255 MiB for all three (serial collector). Exact thresholds depend on the JVM, the collector and the heap settings: with G1 at -Xmx512m on this JDK (1 MiB regions) each of these ~750 KB Strings is a humongous object that takes a whole region, which raises the need here by about a third (measured on the standalone runner, assertions off: the repro parses from -Xmx704m with the serial collector and from -Xmx928m with G1, and fails at 672m and 896m).
Exposure and likelihood261 words
- Who. Applications that pass GeoJSON they do not control to
Polygon.fromGeoJSON, for example to build a polygon filter from a request.Polygonis@lucene.experimental(Polygon.java:41) and advertisesfromGeoJSONin its class javadoc (:25); outside tests (TestPolygon,TestTessellator) no Lucene code calls it. - No setting involved. The parser has no options. The trigger scales with the heap: at 2 GiB the repro parses, and a proportionally longer key or deeper nesting fails there too (an independent reproducer's scaling run: depth 500 under a 200,000-char key fails at 96 MiB and parses at 112 MiB, while the same text as a string value parses at 32 MiB).
- Usual size and depth limits do not prevent it. The repro is 752 KB with 902 levels, below the 1,024-level default that #16423 chose for the expressions parser. A second witness, 1,000 nested property objects with 1,000-char keys (1,005,143 chars), also exhausts 512 MiB (standalone runner, G1). So a 1 MiB input limit plus a 1,000-level depth limit still lets a failing input through. Size and depth limits together do bound the memory, but only when chosen for the heap budget; an application may also need a key-length (or path-length) limit.
- Effect. A VM error instead of the polygon or the declared
ParseException. No wrong polygon is returned and no index is touched. - Realistic input is not at risk. Real GeoJSON is shallow (a MultiPolygon's coordinates are four arrays deep) and uses short keys, so its paths stay short (the formula above); no report in ten years.
One JUnit class, one test method, core only, public API: Polygon.fromGeoJSON on a 751,925-char Feature whose properties hold a 750,000-char key over 900 nested arrays. Deterministic, single thread, default stack, about 2 s. It relies on Lucene's default 512 MiB test heap; with 2 GiB the same input parses.
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.lucene.geo;
import org.apache.lucene.tests.util.LuceneTestCase;
public class TestGeoJSONLongKeyNestedArrays extends LuceneTestCase {
public void testLongPropertyKeyOver900NestedArrays() throws Exception {
String geojson =
"{\"type\":\"Feature\",\"geometry\":{\"type\":\"Polygon\",\"coordinates\":"
+ "[[[0.0,0.0],[1.0,0.0],[1.0,1.0],[0.0,0.0]]]},\"properties\":{\""
+ "k".repeat(750_000)
+ "\":"
+ "[".repeat(900)
+ "]".repeat(900)
+ "}}";
try {
assertEquals(1, Polygon.fromGeoJSON(geojson).length);
} catch (OutOfMemoryError e) {
// Expected: the polygon of this ~750 KB Feature; actual: OutOfMemoryError in a 512 MiB heap.
throw new AssertionError(
"fromGeoJSON of a "
+ geojson.length()
+ "-char Feature ran out of a "
+ (Runtime.getRuntime().maxMemory() >> 20)
+ " MiB heap: "
+ e,
e);
}
}
}
cd $LUCENE && ./gradlew :lucene:core:test --tests TestGeoJSONLongKeyNestedArrays -Ptests.seed=DEADBEEF -Ptests.asserts=true
- Expected
fromGeoJSONreturns one polygon.- Observed
OutOfMemoryError: Java heap spacefromparseArray(path + ".[]", line 309), with assertions on and off.
More files, environment, full output295 words
LuceneTestCase is only the base class; no randomness is used. The test asserts the declared outcome (one polygon) and only turns OutOfMemoryError into a readable message; ParseException is not accepted as a pass. Lucene's default test heap is 512 MiB (tests.heapsize in TestsAndRandomizationPlugin.java:132), so the gradle command needs no heap option.
Regression test: this same class, or its one method added to TestPolygon; it passes with the recommended fix. It depends on the default 512 MiB test heap, and the margin is small: unfixed code parses from about -Xmx704m (serial collector) or -Xmx928m (G1), so a 1 GiB test heap would already hide the bug. A PR should keep the default heap for this test (or pin it).
| lucene | main 645d4fe14a20 (11.0.0-SNAPSHOT): fails; 10.6.0, 10.5.2 and 9.12.3 (this test on the release jars) and releases back to 6.2.0 (a version-portable probe of the same shape): fail; with the fix: passes |
|---|---|
| jdk | OpenJDK 25.0.3 (build 25.0.3+9-LTS) |
| os | Linux, aarch64 (the only platform run) |
cp $REPRO/TestGeoJSONLongKeyNestedArrays.java $LUCENE/lucene/core/src/test/org/apache/lucene/geo/
cd $LUCENE && ./gradlew :lucene:core:test --tests TestGeoJSONLongKeyNestedArrays -Ptests.seed=DEADBEEF -Ptests.asserts=true
# assertions off fails the same way: -Ptests.asserts=false; the default test heap (512 MiB) is part of the repro
# gradle, stock 645d4fe14a20, -Ptests.seed=DEADBEEF -Ptests.asserts=true (trimmed; -Ptests.asserts=false identical)
java.lang.AssertionError: fromGeoJSON of a 751925-char Feature ran out of a 501 MiB heap: java.lang.OutOfMemoryError: Java heap space
at org.apache.lucene.geo.TestGeoJSONLongKeyNestedArrays.testLongPropertyKeyOver900NestedArrays(TestGeoJSONLongKeyNestedArrays.java:40)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.base/java.lang.StringConcatHelper.newArray(StringConcatHelper.java:578)
at java.base/java.lang.StringConcatHelper.doConcat(StringConcatHelper.java:436)
at java.base/java.lang.StringConcatHelper.simpleConcat(StringConcatHelper.java:422)
...
at org.apache.lucene.geo.SimpleGeoJSONPolygonParser.parseArray(SimpleGeoJSONPolygonParser.java:309)
at org.apache.lucene.geo.SimpleGeoJSONPolygonParser.parseArray(SimpleGeoJSONPolygonParser.java:309)
... (every Lucene frame shown is parseArray:309)
:lucene:core:test (FAILURE): 1 test, 1 failure
# "501 MiB" is Runtime.maxMemory() for -Xmx512m with the parallel collector of Lucene's test JVMs;
# the standalone runner (G1) prints "512 MiB" and fails the same way, also with -XX:+UseSerialGC; with -Xmx2g it passes.- The javadoc accepts a Polygon inside a Feature and declares only
ParseException;propertiesmay hold any JSON. - The parser only compares the path with five short constants and a
crsprefix, so the full string is never needed. - #16423 (10.6.0) set the newer bar: parsers should not throw VM errors on nested input.
The full argument193 words
- Contract.
Polygon.java:248-255: "Parses a standard GeoJSON polygon string. The type of the incoming GeoJSON object must be a Polygon or MultiPolygon, optionally embedded under a \"type: Feature\"." It declares onlythrows ParseExceptionand links the GeoJSON spec, where the value of a Feature'spropertiesis any JSON object (or null). - The parser's own example (
SimpleGeoJSONPolygonParser.java:24-38) is a Feature whosepropertieshold a nested object, andTestPolygon.testGeoJSONPolygonFeature(:176) tests that shape. - Properties are meant to be passed over. LUCENE-8964 (#10007, 8.3.0) changed the parser so that string arrays in
propertiesparse instead of failing (TestPolygon.testPolygonPropertiesCanBeStringArrays,:348). - Nothing needs the full path. A 21-char prefix gives the same answer to every path test; a 600,000-document differential confirmed it, and caps of 20 and 12 chars were caught as different.
- Recent maintainer stance. #16423 (10.6.0,
lucene/CHANGES.txt:551): the expressions parser "could throw StackOverflowError on deeply nested expressions ... The new code allows to add a hard limit for nesting (default is 1024) and no longer throws virtual machine errors." - Nothing found treats the growth as intended (the file's history and tests, #8433, #10007, and the prior-report searches below).
The devil's-advocate check183 words
The strongest case for "intended" (built by the intended-behaviour checker and rated moderate by a works-as-designed reviewer, OpenAI GPT-5.6): the parser is deliberately "simplistic" (#8433) and does "minimal parsing" (SimpleGeoJSONPolygonParser.java:47-51), Polygon is @lucene.experimental, nothing promises a memory bound, SECURITY.md leaves limits on untrusted input to the application, the comment at :311 ("This is only used when parsing the \"features\" in type: FeatureCollection") suggests deep content elsewhere was not expected, and LUCENE-6156 (#7218) is a precedent of a maintainer refusing a guard for a recursion error on huge input.
Why it fails: the input is in the documented domain and modest (752 KB; it parses at 2 GiB); the cost comes from a private detail that buys nothing; the usual limits an application would set (about 1 MiB of input, about 1,000 levels) do not prevent it; SECURITY.md itself calls such issues "valid bug reports and hardening suggestions"; LUCENE-8964 answers the :311 comment; and #16423 is eleven years newer than LUCENE-6156. These points bear on severity and labelling, not on whether it is a defect. Verdict: not intended, a minor defect.
Recommended Keep at most the first 21 chars of the path
Cap the path: a childPath() helper keeps at most the first 21 chars, one more than the longest constant compared (features.[].geometry), so every path test gives the same answer and memory stays linear. Validated: the repro passes, and org.apache.lucene.geo (core) gives 194 tests, 0 failures (7 nightly/awaitsfix skipped) with two seeds; the two nightly GeoJSON tessellator tests also pass.
All options and trade-offs3 options · 636 words
One fix was validated in a scratch worktree of 645d4fe14a20; the context-value alternative was not run. A depth guard (the sibling LFV-949's fix) complements it but does not fix this on its own.
A childPath(path, key) helper replaces the four concatenations (:139-153, :309, :312) and keeps at most MAX_PATH_CHARS = 21 chars, one more than the longest constant compared (features.[].geometry, 20 chars). The finder's diff applies unchanged to 645d4fe14a20 (+28/-16 in SimpleGeoJSONPolygonParser.java).
static final int MAX_PATH_CHARS = 21;
/** Returns the (capped) path of a child: key == null means an array element. */
private static String childPath(String path, String key) {
if (path.length() >= MAX_PATH_CHARS) {
return path; // already capped; appending does not change the kept prefix
}
String full;
if (key == null) {
full = path + ".[]";
} else if (path.length() == 0) {
full = key;
} else {
full = path + "." + key;
}
return full.length() > MAX_PATH_CHARS ? full.substring(0, MAX_PATH_CHARS) : full;
}Trade-offs
- Same answers. A path of 21 chars or more equals none of the constants either way, and
startsWith("crs")andisEmpty()depend only on the kept prefix. It must be 21, not 20: at 20,features.[].geometryxwould be cut tofeatures.[].geometry(the differential's cap-20 mutant shows exactly that). - Memory becomes at most 21 chars per level plus the keys themselves, linear in the input. A long key under a short path is still concatenated in full once and then cut (a short-lived copy, not one kept per level).
- Cost of the shortcut: the constant is tied to the longest compared path; a future, longer constant would need the cap raised (the field's javadoc states the 20-char bound it relies on; a PR could add the
features.[].geometryxreason for 21 rather than 20).
What the checks validated
Gradle --no-daemon in a scratch worktree of 645d4fe14a20, OpenJDK 25.0.3 (2026-10-31). Stock: the repro fails with assertions on and off. With the fix:
- the repro passes with assertions on and off;
:lucene:core:test --tests 'org.apache.lucene.geo.*'with seedsDEADBEEFandCAFEBABE: 18 classes, 194 tests, 0 failures, 7 skipped each (TestPolygon16,TestTessellator77,TestSimpleWKTShapeParsing14,TestPolygon2D16, the repro, ...);- one gradle invocation of
checkGoogleJavaFormat,forbiddenApisMainandecjLintMain(core) endedBUILD SUCCESSFUL(the log does not record whether each task ran or was up to date).
Earlier, on a9779ef3c193 (identical file): an independent reproducer's 2-test class passes 2/2 with the cap, and a differential of the original against the capped parser on 600,000 random GeoJSON-like documents gives equal outcomes for all of them while catching cap-20 and cap-12 mutants. On 645d4fe14a20 with the cap shadowed in, the 1,000-nested-objects witness and both controls also return their polygon, assertions on and off (the diagram probe). An independent fact check repeated the geo run (194 tests, 0 failures, 7 skipped) and ran the two skipped nightly tessellator tests that call fromGeoJSON (testComplexPolygon42, testComplexPolygon48) with -Ptests.nightly=true: both pass. Not run: the whole :lucene:core:test.
Pass an enum or int state (root, geometry, features, features.[], features.[].geometry, crs, crs.properties, other) to each child instead of a String.
Trade-offs
Constant memory per level and clearer intent; a larger diff that rewrites every path test, so it needs the same differential to show identical behaviour.
The sibling LFV-949's fix throws ParseException beyond 1,000 levels, as #16423 did for expressions.
Trade-offs
Fixes the StackOverflowError sibling but not this: with only the guard, this repro and the finder's 900-level test still fail with OutOfMemoryError. Ship it together with the path cap.
Applications that build polygons from user-supplied GeoJSON cannot rely on usual request-size and nesting limits here (an input under 1 MiB and 1,000 levels still fails at 512 MiB): until a fix ships they need limits chosen for their heap budget, possibly including a key-length limit.
Semi-formal. A planning probe saw the path memory grow with depth squared at 20,000 nested arrays on a 4 GB thread stack. A Z3 model of the live path strings then found the bounded-depth, long-key input, and heap bisection on the stock jar matched its predictions within 3 MiB. Independent checks confirmed it.
The method in full183 words
Method: semi-formal (an executed probe first, then a solver model that produced the witness).
- Lead. A hunt planner, looking for siblings of the expressions-parser nesting fix #16423, probed the GeoJSON parser:
OutOfMemoryErrorat 20,000 nested arrays (40 KB) with a 512 MiB heap and at 40,000 (80 KB) with 1 GiB, on a 4 GB thread stack, the depth-squared signal of the path strings. On a default stack the siblingStackOverflowErrorcomes first, so that form is not reachable by default. - Model. A hunting unit wrote a Z3 model of the live path strings (closed-form lower bound d(75+K) + 3d(d-1)/2 bytes). Its queries found a witness at depth 1,000 or less (Q1), showed that a 1,024-level depth guard does not bound the memory (Q2), that a path-free fix does (Q3, unsat) and a vacuity check (Q4). This produced the long-key, bounded-depth input that separates it from the recursion bug.
- Validation. Heap-threshold bisection on the stock jar matched the model (predicted 253/252/252/103 MiB, measured 255/255/255/105 MiB). The unit reduced it to a public-API JUnit, which was later minimized to the one-test repro shown here.
No prior report: three 51-query searches (GitHub incl. the JIRA archive, commits, mailing lists, downstream, web), every GeoJSON issue and PR, and the changed files of all 414 open PRs (2026-10-28). Closest: #8433 (introduced the API), #10007 (a properties fix), #16423 (the nesting-limit precedent).
Searches2 searches
Verdict new, meaning the searches below found no public report (keyword search cannot prove absence). Not fixed upstream: the last commit on the parser is 448c7d8e241 (2026-02-11, a toArray refactor); the only open PR touching it, #12295, is an instanceof refactor. Related hunt bugs in the same parser, kept separate: LFV-949 (the StackOverflowError on the same nesting), LFV-562 and LFV-569 (other sites).
| Where | When | How | Found |
|---|---|---|---|
| GitHub issues, PRs and commits (apache/lucene incl. the migrated JIRA archive; Elasticsearch, OpenSearch, Solr; all of GitHub) | 2026-10-28 | the hunt's prior-report tool with three specs (path memory, nesting, GeoJSON general; 16 GitHub, 3 commit, 2 downstream queries each), manual searches for GeoJSON (19 items, all read by title), fromGeoJSON, SimpleGeoJSONPolygonParser, GeoJSON memory / OutOfMemoryError / nesting; the changed files of all 414 open PRs; every issue and PR created since 2026-10-24 | background only: #8433, #10007, #16423, #16549, #12295, #9498, #9584, #8208, #8214; stleary/JSON-java#505 (a similar defect in another library) |
| Mailing lists, JIRA archive and the web | 2026-10-28 | 20 list queries per spec over dev@, java-user@, issues@ and commits@, 4 JIRA queries per spec, 20-21 web queries per spec | only LUCENE-7380, LUCENE-8964, LUCENE-8538 and LUCENE-8452 in JIRA; nothing about this bug |
Born with the API: LUCENE-7380 (#8433, 2016-07-15; 343f374b530, backport 573aaf75f52), first released in 6.2.0. It fails on 13 releases from 6.2.0 to 10.5.2, the 10.6.0 revision and main, with the controls passing; the path code is unchanged since. Not fixed.
Versions tested6 rows
Introduced by LUCENE-7380 (#8433, "Add Polygon.fromGeoJSON"; Mike McCandless, +1 from Robert Muir; JIRA patch, no PR): main-line commit 343f374b530 (2016-07-15, first released in 7.0.0) and its branch_6x backport 573aaf75f52 (same day, first released in 6.2.0). The API and the behaviour arrived together, so no release has one without the other; 6.1.0 has no fromGeoJSON. git log -S on both concatenations, run from 645d4fe14a20, finds only 343f374b530 (with --all also the backport and two whole-tree commits outside main's history that do not change the code); later edits (LUCENE-8964, the 2020 reformat, cbec94c153c, 448c7d8e241) do not touch path building. In 6.x-8.x the failing line is 291, from 9.0.0 it is 309.
| Version or commit | Verdict | Result |
|---|---|---|
main 645d4fe14a20 | affected | minimal repro fails with assertions on and off (gradle and standalone runner) |
10.6.0 (release revision 4ae946e5a9b), 10.5.2, 10.0.0, 9.12.3, 9.0.0 | affected | OutOfMemoryError at parseArray:309; the three controls return the polygon |
| 6.2.0, 6.2.1, 6.6.6, 7.0.0, 7.7.3, 8.0.0, 8.2.0, 8.3.0, 8.11.4 | affected | OutOfMemoryError at parseArray:291; controls pass |
| 6.2.0 jars + 6.2.0's parser with only the path cap | isolation | the bug case returns the polygon (6.2.0's own source, shadowed, still fails) |
| 6.1.0 | no API | no fromGeoJSON (compile error) |
main 645d4fe14a20 with the fix | not affected | repro passes |
Confirmed with medium confidence (implicit contract, contrived input). The finder and independent verifiers reproduced it with their own tests; a works-as-designed review (OpenAI GPT-5.6, reading only) said real; a refute review said not real only because it could not run anything. The adjudication, an audit and this doc's gradle runs re-ran it.
Checks and verdicts457 words
Every executed check ran on stock builds or release jars. No executed check dissented. The earlier cross-model reviews ran reading only; the cross-model fact check is the first that reports an execution.
| Check | Verdict | What it found |
|---|---|---|
| Finder (hunting unit on geo text parsers) | candidate | Own JUnit fails on main (-ea, -da), the 10.6.0 revision, 10.5.2 and 9.12.3; the path cap passes, TestPolygon 16/16 with it, 600,000-document differential equal. |
| Works-as-designed review (OpenAI GPT-5.6, reading only) | real, medium | Intended case moderate but rejected; root cause agrees; could not run Java; a separate re-run failed 1/1. |
| Independent reproducer | real | Own 2-test class, same length and depth: long text as key fails, as value passes, on main, 10.5.2, 9.12.3 (-ea, -da); heap need tracks depth x key length. |
| Refute review (OpenAI GPT-5.6, reading only) | not real, low (procedural) | Could not write or run anything, so its "not real" is procedural; agreed with the root cause and that the input is in contract: "Do not treat this as a substantive refutation". Separate checks failed as claimed. |
| Intended-behaviour devil's advocate | real (low) | Built and rebutted the intended case; added the 1,000-nested-objects witness; probe fails 2 of 4 (controls pass) on main, 10.5.2, 9.12.3. |
| Adjudication | confirmed, medium | Re-ran the independent reproduction's (-da) and the finder's (-ea) tests; file as hardening with the sibling. |
| Registration check and credibility audit | keep | Re-runs on main with -da and -ea; the 1,000-nested-objects witness and both controls as claimed. |
| Minimal repro | verified | One-test class fails on 645d4fe14a20 (-ea, -da, SerialGC, two seeds); passes at 2 GiB and with the path cap. |
| This doc's gradle runs | real; fix validated | Gradle in a scratch worktree of 645d4fe14a20: stock fails with assertions on and off at the default test heap; with the path cap the repro passes and org.apache.lucene.geo (core) gives 194 tests, 0 failures (7 nightly/awaitsfix skipped) with two seeds; the format and lint invocation ended BUILD SUCCESSFUL. |
| Independent fact check | holds; wording fixed | Re-ran the repro (standalone runner and gradle, assertions on and off), the fix, the depth guard alone, its own controls, the 10.6.0, 10.5.2 and 9.12.3 release jars and a heap bisection; found three wording errors (G1 overhead, a control's depth, a test count), all fixed here. |
| Cross-model fact check (OpenAI GPT-5.6) | real; wording fixed | Reports a small in-memory probe on the 645d4fe14a20 jar: the witness ran out of 512 MiB, the short-key and flat controls returned 1 polygon. Its wording points (input limits, a control's depth, the git log -S result, how properties are parsed, the source of the shown output) are fixed here. |
The diagram as a still image, generated with AI and checked against the code. It is under 10 MB, the limit for a GitHub attachment.
- PNG 531 KB
Finding record, Lucene and upstream sources, repro files and diagram notes1,612 words
- Registry title
- Polygon.fromGeoJSON keeps a full key-path String alive at every nesting level, so its heap use grows with nesting depth times key length: a valid 750 KB GeoJSON Feature (900 nested arrays under one long property key), or a 1 MB one (1,000 nested objects with 1,000-char keys), runs out of a 512 MiB heap (every release since 6.2.0)
- Area
- core / geo GeoJSON parsing (Polygon.fromGeoJSON, SimpleGeoJSONPolygonParser)
- Priority
- Low, rank 824: An OutOfMemoryError from a hostile but valid nested GeoJSON document in the experimental fromGeoJSON is a resource failure from the same adversarial-nesting trigger as LFV-949.
- Impact class
- minor
- Impact note
- Hits only applications that pass GeoJSON text to Polygon.fromGeoJSON (core; Polygon is @lucene.experimental; no non-test caller in the Lucene tree) with a valid Feature whose properties combine hundreds of nesting levels with long keys: the heap need is about nesting depth x key length (d*K + 1.5*d^2 bytes), so 900 nested arrays under a 745,654-char key (747,598 chars of text) or 1,000 nested objects with 1,000-char keys (1,005,143 chars) exhaust a 512 MiB heap with OutOfMemoryError: Java heap space, roughly 500 to 1,000 times the input size, while the same text as a string value, a flat array under the long key, or 900 levels under a short key all parse; no setting is involved, ordinary request-size caps and a 1,000-level depth cap do not bound it, realistic GeoJSON (shallow, short keys) is unaffected, no wrong polygon is returned and no index is touched; thresholds depend on the collector (SerialGC bisection; G1 humongous strings need about twice as much); with -ea and -da, in every release from 6.2.0 to 10.6.0 and on main.
- Summary
- You call Polygon.fromGeoJSON on a valid GeoJSON Feature whose properties hold one member with a very long key (about 745,000 chars) above 900 nested arrays, or 1,000 nested objects each under a 1,000-char key (about 1 MB of text), and expect the polygon back. Instead the call fails with OutOfMemoryError: Java heap space in a 512 MiB heap, because SimpleGeoJSONPolygonParser builds a new full key path (path + "." + key, path + ".[]") for every nesting level and each recursive frame keeps its copy alive, so live memory grows with depth times path length, although the parser only ever compares the path with five short constants and a "crs" prefix. The same text as a string value, or the same nesting under a short key, parses fine, so neither a size cap nor a usual depth cap protects the caller.
- Introduced
- 343f374b530fa71dc6102d74725b536f5f1367f3 (main line, author date 2016-07-15, 'LUCENE-7380: add Polygon.fromGeoJSON' by Mike McCandless, +1 from Robert Muir; LUCENE-7380 is apache/lucene#8433; JIRA patch era, no pull request; first shipped in 7.0.0) and its branch_6x backport 573aaf75f52f446c8b7ab915eefd420013c544a1 (same author date, shipped in 6.2.0). The per-level path building was born with the parser and is not a regression [read: git log -S on both concatenations finds only these two commits plus two whole-tree branch commits (527c6f350c5, aad44bc6e36) that do not change the code; git blame -w -M -C -C on main attributes every path-building line to 343f374b530 (the isEmpty() checks at :197 and :223 to cbec94c153c, equals("") -> isEmpty(), no semantic change); exec: on the 6.2.0 jars, shadowing 6.2.0's own parser source unmodified still fails, and the same source with only the candidate's path cap back-ported passes] (2016-07-15 (author date of 343f374b530 and 573aaf75f52, 14:51:08Z; the backport was committed at 14:51:35Z; JIRA resolved the same day))
- First affected version
- 6.2.0 [exec] (the first release with Polygon.fromGeoJSON and SimpleGeoJSONPolygonParser; 6.1.0 has Polygon but no fromGeoJSON [exec: compile error]); first 7.x release 7.0.0 [exec], from the main-line commit
- Affected versions
- Every release from 6.2.0 to 10.6.0 (10.6.0 run as its release revision, branch_10_6 head 4ae946e5a9b) and main a9779ef3c193. [exec] the version-portable probe of the version-history runs (900 nested arrays under a long key, 512 MiB heap, three controls) fails with OutOfMemoryError at parseArray's path + ".[]" (line 291 in 6.2.0-8.11.4, 309 from 9.0.0) while all three controls return the polygon on 6.2.0 (-da, -ea, and -da with SerialGC), 6.2.1, 6.6.6, 7.0.0, 7.7.3, 8.0.0, 8.2.0, 8.3.0, 8.11.4, 9.0.0, 9.12.3, 10.0.0, 10.5.2, branch_10_6 and main (-da, and -da with SerialGC); the candidate JUnit (an old-package copy before 9.1) fails 1 of 1 with -ea on 6.2.0, 8.11.4, 9.0.0, 9.12.3, 10.5.2, branch_10_6 and main, and with -da on main; the finder's repro.sh runs fail on main (-ea and -da), branch_10_6, 10.5.2 and 9.12.3 (-da); the independent reproduction's JUnit fails 1 of 2 (control passes) on main, 10.5.2 and 9.12.3 (-ea and -da); the intended-behaviour check's probe fails 2 of 4 (both controls pass) on main (-ea and -da), 10.5.2 and 9.12.3 (-da); separate checks fail on main (-ea and -da) and 10.5.2 (-da); the adjudication re-ran on main (-ea and -da) and the registration check on main (-da and -ea). [read] the path-building code is unchanged from 343f374b530 to main; the releases not executed are inferred. Not fixed upstream: upstream main 645d4fe14a20 (2026-10-28) has no commit on SimpleGeoJSONPolygonParser.java after 448c7d8e2414 (author date 2026-02-11, a toArray refactor). All runs on Linux aarch64 with JDK 25.0.3; the heap thresholds come from the JVM's String layout and collector, not from the Lucene version; x86 was not measured.
- Status
- documented (brief, checks and a per-finding doc exist)
- Verification
- Confirmed by AI-run checks: an independent repro, a cross-model review that could only read the code (its low-confidence “not real” is procedural), an intended-behaviour check, a prior-report search and a skeptical audit; not yet triaged by a person.
- Found by
- semi-formal; also reached by Z3
- Upstream and prior reports
- none found (the prior-report record, verdict new, searched 2026-10-28, read-only): three scripted prior-report searches (51 queries each: 16 GitHub issue/PR, 3 commit, 2 downstream, 20 mailing-list over dev@, java-user@, issues@ and commits@, 4 JIRA-archive and 20-21 web; 269, 246 and 280 distinct hits, 0 high) plus the finder's run (51 queries, 188 hits, 1 high = #8208, unrelated) and the intended-behaviour check's leg (163 hits), manual GitHub searches (every apache/lucene issue or PR mentioning GeoJSON (19) triaged; fromGeoJSON, SimpleGeoJSONPolygonParser, GeoJSON memory, OutOfMemoryError and nesting phrasings; GitHub-wide and in Elasticsearch, OpenSearch and Solr), the changed files of all 414 open apache/lucene PRs (only #12295, an instanceof refactor, touches the parser), every apache/lucene issue and PR created since 2026-10-24, the dev@ thread of 2026-10-28, and 20 web queries: nothing reports or discusses per-level path strings, memory growth with nesting depth or key length, or an OutOfMemoryError in Polygon.fromGeoJSON or SimpleGeoJSONPolygonParser. Closest items, not reports of this bug: #8433 (LUCENE-7380), the introducing issue, no memory or depth discussion; #10007 (LUCENE-8964, 8.3.0), the only later parser fix (skip string arrays in properties); #16423 (merged 2026-07-27, 10.6.0), the precedent for a nesting limit that throws ParseException, in lucene/expressions only; #8208 (LUCENE-7153), an OutOfMemoryError from polygon-query bitsets; stleary/JSON-java#505, an analogous small-input OutOfMemoryError in another JSON library. Not fixed: no commit on the parser upstream after 448c7d8e2414 (2026-02-11). Refresh 2026-10-28 (GitHub search and REST API, read-only): every apache/lucene PR created since 2026-10-26 (through #16936, created 2026-10-28; the file lists of the code PRs touch no geo file) and every issue created since 2026-10-23 (through #16937) concern no geo code; 'GeoJSON' (19 items, newest #15205 from 2025-09), 'SimpleGeoJSONPolygonParser' (#9584, #9498), 'fromGeoJSON' (#8433 only), 'GeoJSON memory' and 'GeoJSON OutOfMemoryError' (#8208 only) return nothing new; upstream main 645d4fe14a20 has no newer commit on the file.
- Lucene commit tested
- main
645d4fe14a20(2026-10-28): the minimal repro fails with assertions on and off through gradle in a scratch worktree at the default test heap, and through a standalone runner (G1, also with the serial collector); with the path cap it passes. In the same worktree, with the fix, the tests oforg.apache.lucene.geo(core) give 194 tests, 0 failures (7 nightly/awaitsfix skipped) with two seeds (2026-10-31). An independent fact check re-ran the repro on main and on the 10.6.0, 10.5.2 and 9.12.3 release jars: it fails on all of them. The hunt's earlier checks ran on maina9779ef3c193, whereSimpleGeoJSONPolygonParser.javaandPolygon.javaare identical, and on releases 6.2.0 to 10.6.0.
- Finding record
lucene/core/src/java/org/apache/lucene/geo/SimpleGeoJSONPolygonParser.java:47-51 (at a9779ef3c193) ('Does minimal parsing'), :99 (path comment: 'the "address" by keys'), :100 (parseObject), :138-153 (newPath = path + "." + key at :143 and :151, recursion at :145 and :153), :125, :174, :186, :192-197, :205, :222-223 (every use of path: equals with short constants, startsWith("crs"), isEmpty), :283, :309 and :312 (parseArray passes path + ".[]")lucene/core/src/java/org/apache/lucene/geo/Polygon.java:41 (at a9779ef3c193) (@lucene.experimental), :248-257 (fromGeoJSON javadoc: a Polygon or MultiPolygon 'optionally embedded under a "type: Feature"'; throws ParseException); in-tree callers: only TestPolygon and TestTessellator (grep, registration check)
- The finding in 30 seconds
a Lucene main checkout (645d4fe14a20) lucene/core/src/java/org/apache/lucene/geo/SimpleGeoJSONPolygonParser.java:65, :99-100, :125, :138-153, :174, :186, :192-197, :205, :222-224, :283, :307-313
- Who is affected
a Lucene main checkout (645d4fe14a20) lucene/core/src/java/org/apache/lucene/geo/Polygon.java:25, :41, :248-256; grep of lucene/ for fromGeoJSON (only TestPolygon and TestTessellator outside Polygon.java)a Lucene main checkout (645d4fe14a20) lucene/CHANGES.txt:551 (GITHUB#16423, default nesting limit 1024)
- Minimal repro
a Lucene main checkout (645d4fe14a20) build-tools/build-infra/src/main/java/org/apache/lucene/gradle/plugins/java/TestsAndRandomizationPlugin.java:132 (tests.heapsize 512m)
- Why it looks unintended
a Lucene main checkout (645d4fe14a20) lucene/core/src/java/org/apache/lucene/geo/Polygon.java:41, :248-256a Lucene main checkout (645d4fe14a20) lucene/core/src/java/org/apache/lucene/geo/SimpleGeoJSONPolygonParser.java:24-38, :47-51, :311a Lucene main checkout (645d4fe14a20) lucene/core/src/test/org/apache/lucene/geo/TestPolygon.java:176, :348a Lucene main checkout (645d4fe14a20) lucene/CHANGES.txt:551SECURITY.md:38, :46
- History
a Lucene clone: releases/lucene/10.6.0 = 4ae946e5a9b
- TestGeoJSONLongKeyNestedArrays.java (one test class):
repro/LFV-952/src/org/apache/lucene/geo/TestGeoJSONLongKeyNestedArrays.java(46 lines)
Made in parallel with this doc from the code at 645d4fe14a20 and the executed repro runs.
assets/LFV-952/bug.png(531 KB)
![Diagram of the bug: a valid GeoJSON Feature whose properties hold one 750,000-char key over 900 nested arrays. At every nesting level the parser keeps a new, longer copy of its key path (properties.kkk… followed by .[] per level); 900 copies total about 676 million chars, about 645 MiB, so a 512 MiB heap runs out. With the path capped at 21 chars the same input parses.](../assets/LFV-952/bug.png)