XML query parser throws StackOverflowError instead of ParserException on deeply nested queries

CoreParser turns query XML into a Lucene Query by recursion: each compound builder (BooleanQuery, DisjunctionMaxQuery, ConstantScoreQuery and the span builders) calls back into QueryBuilderFactory or SpanQueryBuilderFactory for its children, so every nesting level adds two Java frames, and nothing counts them. Well-formed XML from the documented vocabulary, nested a few thousand levels deep, therefore throws java.lang.StackOverflowError out of parse(InputStream), getQuery(Element) and getSpanQuery(Element), which declare only ParserException. Whether the text entry gets that far depends on the JDK's XML parser: Java 21, the 10.x minimum, has no element-depth limit by default, so on 10.6.0 plain parse(InputStream) overflows at 3,000 nested BooleanQuery levels (about 180 KB) on a 1 MB stack and 5,000 (about 300 KB) on the 2 MB default; Java 25, main's minimum, rejects text deeper than 100 elements with a ParserException (tested builds: OpenJDK 21.0.10 and 25.0.3). No JDK XML setting applies to the DOM entries; they overflowed on every JDK tested. Nothing else goes wrong: no query is returned, no index effect, and the JVM continues. The recursion dates from the parser's original contribution (2006; in the 1.9 release tag, first published as a jar in 1.9.1). A nesting limit in CoreParser, the approach #16423 took for the expressions parser in 10.6.0, fixes it; the repro and the 69 existing XML-parser tests pass with it.

• CoreParser: the XML query parser in lucene-queryparser (org.apache.lucene.queryparser.xml), with the subclasses CorePlusQueriesParser and CorePlusExtensionsParser. Its element vocabulary is documented in LuceneCoreQuery.dtd.
• Entry points: parse(InputStream) (XML text), and the public getQuery(Element) / getSpanQuery(Element) (a DOM element the caller supplies). All three declare only ParserException.
• Element-depth limit: the JDK's XML parser setting jdk.xml.maxElementDepth. In the tested OpenJDK 25.0.3 build, conf/jaxp.properties:184 sets it to 100; in the tested 21.0.10 build, line 163 is commented out (# jdk.xml.maxElementDepth=0), so there is no limit (other builds were not checked).

1. CoreParser.java:141-143: parse(InputStream) returns getQuery(parseXML(xmlStream).getDocumentElement()). parseXML (:182-206) turns off validation (:184), enables FEATURE_SECURE_PROCESSING (:186; a ParserConfigurationException there is ignored, :187-192) and wraps every Exception from creating the parser or parsing the text in a ParserException (:196-198, :203-205); it catches Exception, not Error. Lucene sets no depth limit of its own.
2. CoreParser.java:208-216: getQuery(Element) and getSpanQuery(Element) hand the element straight to queryFactory / spanFactory, with no depth state.
3. QueryBuilderFactory.java:29-35 looks up the builder by element name and calls it (:34). BooleanQueryBuilder.java:57 calls factory.getQuery(clauseQuery) for each clause; DisjunctionMaxQueryBuilder.java:54 and ConstantScoreQueryBuilder.java:41 do the same, and the span builders call SpanQueryBuilderFactory.java:46 (SpanOrBuilder.java:41, SpanNearBuilder.java:45, SpanFirstBuilder.java:38, SpanNotBuilder.java:42-43, SpanPositionRangeBuilder.java:39).
4. Each nesting level adds a builder frame and a factory frame. The executed trace cycles BooleanQueryBuilder.getQuery(BooleanQueryBuilder.java:57) and QueryBuilderFactory.getQuery(QueryBuilderFactory.java:34) until StackOverflowError.
5. Two more recursions, found by the hunt's static model and exercised only in its sweep: DOMUtils.getTextBuffer (DOMUtils.java:157-177, once per nested element at :167 and per entity reference at :172) and DOMUtils.getAttributeWithInheritance (:88-101, once per ancestor). These four cycles are the XML module's own; the classic QueryParser that UserInputQueryBuilder calls (UserInputQueryBuilder.java:59-73) and custom builders are outside the model.

Rows 1-4 come from the hunt's 152-JVM sweep: one JVM per shape, entry, stack and JDK, trying depths 50, 100, ..., 2,000, 3,000, 5,000, 7,500, ... in ascending order until the first overflow. The 1,000 / 3,000 / 5,000 / 10,000-level points of rows 1 and 3 were also run as cold single calls, one call per fresh JVM, with the same outcome, and row 5 comes from those cold single calls only; the audit probe (ascending, one JVM) agrees on row 3. The fact check re-ran row 1 on the 10.6.0 release jars: at 1 MB, 2,000 levels parsed and 3,000 overflowed 6 of 6 times each. The main rows ran on a9779ef3c193; no commit touches the xml module between it and 645d4fe14a20. Warm JVMs need several times more levels (the independent reproduction: 1,933 to 24,872 on the 10.6.0 commit's jars with Java 21, 256 KB to 2 MB stacks). x86_64 was not measured.

• 10.x and 9.x on Java 21: the plain parse(InputStream) of well-formed XML, default settings (executed on the 10.6.0 commit, 10.5.2 and 9.12.3; 9.12.3 also on Java 17).
• Any JDK (no JDK XML setting applies to a DOM): getQuery(Element) (executed on Java 17, 21 and 25) / getSpanQuery(Element) (21 and 25) with a DOM the application builds (a 2019 java-user@ thread shows applications doing this).
• Main: also with another JAXP implementation on the classpath (xercesImpl 2.12.2, executed) or a relaxed jdk.xml.maxElementDepth (executed with 0).
• Default vs opt-in: the XML query parser is an opt-in module class; the trigger needs no non-default setting.
• Likelihood: contrived. On 1-2 MB stacks it takes about 3,000-5,000 levels; only an adversary or a broken generator produces that.
• Effect: the calling thread gets java.lang.StackOverflowError, which catch (ParserException) and catch (Exception) handlers miss. No query is returned, no index or data is touched, and the JVM and later parses continue (CoreParser keeps no per-parse state; QueryBuilder.java:22-26 requires thread-safe builders).
• The parser fails first: a nest of one clause per level around one term still searches where the parser already overflows (1 MB stack: 2,000 levels parse and IndexSearcher.count returns 1, 3,000 overflow in the parser; 2 MB: 5,000 parse and search, 10,000 overflow in the parser). Core search is not unbounded either: IndexSearcher.count and toString of the same tree built in Java overflowed at 5,000 and 7,500 levels on a 1 MB stack (JIT-dependent), so the parser is the earliest limit, not the only one.

It uses the DOM entry because that fails on main, where Java 25's XML depth limit guards the text path. 100,000 levels give a wide margin over the 3,000-10,000 needed on 1-2 MB stacks; on a much larger thread stack the stock code could return a Query and the test would pass (not tried). The whole test method, DOM build and failing call, runs in about 0.1-0.3 s (gradle test reports). ./gradlew tidy re-wraps the one-line catch (@SuppressWarnings("unused") ParserException expected) onto three lines; nothing else changes. The line numbers in the observed trace are those of the file as shipped.

On 10.x the text path behaves as follows (no second test needed): on 10.6.0 with Java 21 and -Xss1m, parse(InputStream) of 3,000 nested BooleanQuery levels (about 180 KB) throws StackOverflowError, while Java 25 gives ParserException from the JDK's depth limit. The hunt's and the independent reproduction's runs swapped only the JDK binaries in the harness's generated command (the harness is pinned to Java 25); the fact check re-ran it with its own probe on the 10.6.0 release jars and OpenJDK 21.0.10: 2,000 levels parse and 3,000 overflow, 6 of 6 times each.

Regression test: this same class (or its one method added to TestCoreParser). It passes with the recommended fix and asserts only the contract (a Query or ParserException), not a particular limit. On its own it does not show whether the fix's depth check or its fallback fired; the validation-only 511/512 method covers the boundary.

cp $REPRO/TestCoreParserDeepNesting.java $LUCENE/lucene/queryparser/src/test/org/apache/lucene/queryparser/xml/
cd $LUCENE && ./gradlew :lucene:queryparser:test --tests TestCoreParserDeepNesting -Ptests.seed=DEADBEEF -Ptests.asserts=true
# assertions off fails the same way: -Ptests.asserts=false

# gradle, stock 645d4fe14a20, -Ptests.seed=DEADBEEF -Ptests.asserts=false (test-result XML; stack trimmed)
java.lang.AssertionError: expected a Query or ParserException, got StackOverflowError
	at __randomizedtesting.SeedInfo.seed([DEADBEEF:176E3F6D1EC607DC]:0)
	at org.apache.lucene.queryparser.xml.TestCoreParserDeepNesting.testDeeplyNestedBooleanQuery(TestCoreParserDeepNesting.java:45)
Caused by: java.lang.StackOverflowError
	at java.base/java.lang.StringLatin1.compareTo(StringLatin1.java:114)
	...
	at org.apache.lucene.queryparser.xml.builders.BooleanQueryBuilder.getOccursValue(BooleanQueryBuilder.java:71)
	at org.apache.lucene.queryparser.xml.builders.BooleanQueryBuilder.getQuery(BooleanQueryBuilder.java:54)
	at org.apache.lucene.queryparser.xml.QueryBuilderFactory.getQuery(QueryBuilderFactory.java:34)
	at org.apache.lucene.queryparser.xml.builders.BooleanQueryBuilder.getQuery(BooleanQueryBuilder.java:57)
	at org.apache.lucene.queryparser.xml.QueryBuilderFactory.getQuery(QueryBuilderFactory.java:34)
	... (the last two frames repeat to the end of the recorded trace)

:lucene:queryparser:test (FAILURE): 1 test, 1  failure

# -Ptests.asserts=true: the same failure (1 test completed, 1 failed). The frame where the stack runs out varies
# between runs. The standalone runner gives the same message on main (-ea, -da) and on the 10.6.0 and 9.12.3 jars (-da).

• Declared exception. parse(InputStream), getQuery(Element) and getSpanQuery(Element) declare only ParserException (CoreParser.java:141, :209, :214), documented as "Thrown when the xml queryparser encounters invalid syntax/configuration" (ParserException.java:19); parseXML wraps every Exception from the XML parser (CoreParser.java:196-205; it catches Exception, so an Error is not wrapped). This is an expectation from the throws clause, not a @throws guarantee: parse has no @throws text (CoreParser.java:137-140), and a checked throws clause promises nothing about Errors.
• Documented input. LuceneCoreQuery.dtd:54-57: "Many of the elements can nest other elements to produce queries/filters of an arbitrary depth and complexity. The BooleanQuery element is one such example ... (including other BooleanQueries)". The DTD documents the vocabulary; it is not a runtime contract, since CoreParser does not validate against it (dbf.setValidating(false), CoreParser.java:184). The upstream XML tests check only shallow nesting.
• JDK-dependent outcome. For one input and one Lucene release, the result flips between ParserException (Java 25) and StackOverflowError (Java 21). That points to an accident, not a design.
• Precedent. #16423 (uschindler, approved by rmuir and msokolov; 10.6.0) added a nesting limit of 1,024 and a StackOverflowError fallback to the expressions parser; lucene/CHANGES.txt:551-555 says it "no longer throws virtual machine errors" (JavascriptCompiler.java:104-105, :138, :266-271). On #11687 (SimpleQueryParser stack overflow, open since 2022) mikemccand asked "I wonder if our other query parsers have this problem too?"; nobody answered.
• SECURITY.md:46: issues outside the security scope "are still valid bug reports and hardening suggestions" that can be filed publicly.

The strongest case for "intended" (rated strong by reading-only cross-model lenses, moderate by the intended-behaviour check): (1) a throws clause says nothing about VirtualMachineErrors, and deep nesting is valid syntax, so "invalid syntax/configuration" arguably does not apply; (2) SECURITY.md:38 names "deeply nested queries" as input the application must bound, and SECURITY.md:42 trusts caller-supplied data structures such as a DOM; (3) on #752 rmuir wrote "If you hit StackOverFlowError, really you should let the VM exit", and on #16423 uschindler wrote "The stopped JVM is actually not a bug in Lucene" (#16423 is labelled an enhancement); (4) on main the stock text path is already guarded by the JDK 25 limit; (5) core search also overflows on deep trees; (6) #16820 proposes deprecating the XML parser for Lucene 11 ("mainly a source of security issues", romseygeek).

Why it fails: none of this makes the Error the documented behaviour of a method whose only declared failure is ParserException, for well-formed input the DTD invites; the result for one input flips with the JDK; and the same reviewers approved a nesting limit for the expressions parser in #16423, whose description says "this should be handled". On #752 msokolov wrote "we should not be catching Error" but "If we could somehow detect the problem before it gets to that, then throwing IAE would make sense", which is what the recommended fix does with the parser's own exception. Points (2) and (6) settle the security rating (none), not whether it is a bug.

One fix was validated in a scratch worktree of 645d4fe14a20; the alternatives were not run on their own. Whether a fix is wanted at all, given the proposal to deprecate the XML parser for Lucene 11 (#16820), is the maintainers' call; 10.6 is the last 10.x minor, so a 10.x fix would go into a 10.6.x bugfix release.

The change is +73/-3 lines in CoreParser.java and LuceneCoreQuery.dtd (the hunt's sketch with wording edits, formatted with ./gradlew tidy): a public DEFAULT_MAX_NESTING_DEPTH = 1024 and a protected getMaxNestingDepth(); getQuery(Element) / getSpanQuery(Element) first run an iterative check of the element's ancestors and subtree that throws ParserException("Query nesting level too deep (maximum 1024 levels)"), then call the factory inside a StackOverflowError catch that throws ParserException. The DTD's "arbitrary depth" sentence now names the limit. The same shape uschindler used in #16423.

  @Override
  public Query getQuery(Element e) throws ParserException {
    checkNestingDepth(e);
    try {
      return queryFactory.getQuery(e);
    } catch (StackOverflowError soe) {
      // the depth check should catch this before, but a large limit or a small stack may not
      throw new ParserException("Query nesting level too deep", soe);
    }
  }

Trade-offs

• Deterministic once a DOM reaches getQuery(Element) / getSpanQuery(Element), whatever the JDK or JAXP implementation, with the parser's own exception; overridable like the expressions limit. On the text path the JAXP parser still builds the DOM first, and an Error thrown there (an OutOfMemoryError, say) is not converted. The fallback turns any StackOverflowError from a builder, a custom builder's included, into "nesting too deep".
• The check counts the depth in DOM nodes (elements, text and comment nodes, and the ancestors of the element passed in; not attributes), a leaf's text node included: 511 nested BooleanQuery levels (1,024 nodes) are accepted, 512 rejected. So the "1024 levels" in the message and the DTD sentence means nodes. On Java 25 the JDK already rejects text deeper than 100 elements, so main's text path does not change; on Java 21, queries from 512 levels up to today's overflow point (2,000-5,000 levels cold on 1-2 MB stacks, more when warm: the independent reproduction measured up to about 24,000) would now be rejected. Counting builder levels instead is a small change.
• One extra iterative pass over the whole DOM per parse, even for a large flat query; its cost was not measured.
• 1,024 nodes can still overflow a very small stack (500 BooleanQuery levels overflow at -Xss256k in the hunt's sweep), hence the fallback. On #752 rmuir and msokolov objected to catching StackOverflowError; #16423 later kept such a catch behind its limit. The catch can be dropped.
• Covers CoreParser and its subclasses, not direct use of the public QueryBuilderFactory, the builders or the internal DOMUtils.
• New public API (a constant and a protected method), which could stay private; the change also needs a CHANGES entry.
• The regression test asserts only the contract, so it does not show which of the two mechanisms fired; the validation-only 511/512 method covers the boundary.

What the checks validated

• Gradle --no-daemon, scratch worktree of 645d4fe14a20, OpenJDK 25.0.3, 2026-10-31. Stock: the repro fails with assertions on and off.
• With the fix: the repro passes with assertions on and off; :lucene:queryparser:test --tests 'org.apache.lucene.queryparser.xml.*' passes 73 tests, 0 failures, with seeds DEADBEEF and CAFEBABE: the 69 upstream tests (TestCoreParser 22, TestCorePlusQueriesParser 23, TestCorePlusExtensionsParser 24), the repro, and 3 validation-only methods (511 levels parse, 512 get the depth ParserException; 100,000 nested SpanOr through getSpanQuery get it too; with the limit raised to Integer.MAX_VALUE, 100,000 levels get a ParserException caused by StackOverflowError).
• checkGoogleJavaFormat, forbiddenApisMain, ecjLintMain and renderJavadoc of :lucene:queryparser pass.
• Earlier, the hunt's sketch passed the hunt's tests on main (-ea, -da, with xercesImpl) and on the 10.6.0 commit with Java 25 and 21, and the independent reproduction's 4-test class. Not run: the whole :lucene:queryparser:test.

Wrap the factory calls in the catch above, without a limit.

Trade-offs

Smallest change, but where it fails still depends on stack size and JIT state, and it is what rmuir and msokolov objected to on #752. Not run on its own (the third validation-only method exercises the same catch).

Track the depth per call and throw ParserException past the limit.

Trade-offs

Also covers direct use of the factories, but needs per-call state (one CoreParser may parse on many threads at once, QueryBuilder.java:22-26), for example a ThreadLocal or a changed builder signature, and adds work to every builder call. Not run.

Say in the javadoc and the DTD that callers must bound nesting (for text, with jdk.xml.maxElementDepth).

Trade-offs

No code change, but it does not help the DOM entries or other JAXP implementations, and the result for one input still depends on the JDK. Not run.

Method: non-formal (the first signal was an executed probe; the model and the sweep came after, to characterise it).

• Lead. The hunt planners asked whether the XML query parser is a sibling of the expressions nesting fix (#16423) once the Java 25 XML limit is out of the way. Their main() probes printed StackOverflowError for CoreParser.parse(InputStream) at 5,000 and 20,000 nested BooleanQuery levels on the 10.6.0 commit's jars with Java 21 (1,000 parsed), for parse on main with xercesImpl on the classpath, and for getQuery(Element) on main at 20,000. That overturned an earlier hunt's negative, which had tested Java 25 only.
• Model. A static recursion model over the real bytecode (class-hierarchy call graph, Tarjan strongly connected components, guard ranges and a depth-state detector) found four stateless, unguarded recursive cycles in the XML module's own code, reachable from parse, getQuery and getSpanQuery (the classic QueryParser that UserInputQueryBuilder calls, and custom builders, are outside it); its vacuity probes were classified correctly, and the CoreParser entries flip to safe with the fix.
• Sweep. 19 shape and entry pairs (10 shapes x 2 entries, minus the DOM-only inherit shape's text entry) x 4 stack sizes x {Java 21 + 10.6.0 commit jars, Java 25 + main jars} = 152 JVMs. Each tried depths from 50 upward in ascending order and stopped at the first overflow, giving the last passing and first overflowing grid point per combination.
• Checked by an independent reproducer, a devil's-advocate check, reading-only cross-model reviews, an adjudication, a credibility audit and a noise audit (see below).