XML query parser rejects a <Clause> without occurs instead of treating it as SHOULD

In the XML query parser (CoreParser, CorePlusQueriesParser, CorePlusExtensionsParser), the occurs attribute of a <Clause> says whether the clause is optional (should), required (must), excluded (mustNot) or a filter. The shipped LuceneCoreQuery.dtd declares <!ATTLIST Clause occurs (should | must | mustnot) "should">, its generated docs say "Default value: should", and BooleanQueryBuilder.getOccursValue has its own branch for a missing attribute. Yet CoreParser.parse of <BooleanQuery fieldName='f'><Clause><TermQuery>a</TermQuery></Clause></BooleanQuery> throws ParserException: Invalid value for "occurs" attribute of clause:. org.w3c.dom.Element.getAttribute returns "" for an absent attribute, so the occs == null branch is dead and "" falls through to the throw; occurs="" fails the same way. LUCENE-6270 introduced it in 5.1.0 while adding occurs="filter": the rewrite replaced the original "".equals(occs) test with occs == null. The parse fails cleanly, with no wrong results; also treating an empty value as SHOULD fixes it.

• XML query parser: org.apache.lucene.queryparser.xml in lucene-queryparser. CoreParser registers a BooleanQueryBuilder for <BooleanQuery> (CoreParser.java:93 at 645d4fe14a20); CorePlusQueriesParser and CorePlusExtensionsParser inherit it.
• Element.getAttribute: the DOM call the builders use. Its JDK javadoc: "The Attr value as a string, or the empty string if that attribute does not have a specified or default value."
• DTD default: <!ATTLIST Clause occurs (should | must | mustnot) "should"> (LuceneCoreQuery.dtd:124). An XML parser fills such a default in only when it reads the DTD; CoreParser parses without validation (CoreParser.java:184) and its default entity resolver refuses external entities (:169-171, :218-226), so a document cannot pull the shipped DTD in.

static BooleanClause.Occur getOccursValue(Element clauseElem) throws ParserException {
  String occs = clauseElem.getAttribute("occurs");                      // :71, "" when absent
  if (occs == null || "should".equalsIgnoreCase(occs)) {                // :72, null never happens
    return BooleanClause.Occur.SHOULD;
  } else if ("must".equalsIgnoreCase(occs)) {
    return BooleanClause.Occur.MUST;
  } else if ("mustNot".equalsIgnoreCase(occs)) {
    return BooleanClause.Occur.MUST_NOT;
  } else if ("filter".equals(occs)) {
    return BooleanClause.Occur.FILTER;
  }
  throw new ParserException("Invalid value for \"occurs\" attribute of clause:" + occs);  // :81
}

1. BooleanQueryBuilder.getQuery calls getOccursValue for each <Clause> (BooleanQueryBuilder.java:54).
2. With no occurs, getAttribute returns "" (:71); a DOM built like CoreParser's gives hasAttribute("occurs") == false and getAttribute("occurs") == "".
3. occs == null (:72) is false and "" matches none of the four names, so :81 throws, with the empty value after the colon. The exception leaves CoreParser.parse (CoreParser.java:141-142), which declares ParserException; the caller gets no query.

The same builder reads its other two attributes correctly, through DOMUtils.getAttribute(e, name, default), which treats null or empty as "use the default" (DOMUtils.java:119-137): minimumNumberShouldMatch (:46, default 0) and boost (:63, default 1.0), the defaults the DTD declares (LuceneCoreQuery.dtd:115, :109). Only occurs is read raw and tested for null.

In 5.x from 5.1.0 on, <BooleanFilter><Clause> without occurs was rejected the same way (run on 5.1.0 and 5.5.5); that builder is gone since 6.0.0.

• Who is hit. Applications that use CoreParser, CorePlusQueriesParser or CorePlusExtensionsParser (public, in the exported packages org.apache.lucene.queryparser.xml and .builders) and send a <Clause> without occurs, or with occurs="": hand-written queries, query templates (the DTD names XSLT query templates as a motivation) or generators that rely on the documented default. Solr's xmlparser query parser reaches the same builder (code reading, not run).
• Default vs opt-in. The XML query parser is opt-in; inside it no setting is needed, the default CoreParser(String, Analyzer) constructor is enough. Assertions on or off make no difference.
• What happens. The whole query fails with a checked ParserException; nothing is built or searched, so there are no wrong results, and the same parser instance keeps working.
• Scale. None: one clause triggers it.
• Likelihood: rare. All 28 <Clause> elements in the upstream test fixtures and the DTD's own example write occurs out; a hit fails loudly during development; no public report of it since 5.1.0 (2015) was found.
• Workaround. Write occurs="should" on every optional clause.

Pure XML parsing: no index and no randomness (TermQuery does not analyze; the seed does not matter). The ParserException is rethrown as an AssertionError that states expected and actual, with the original exception as its cause. It compiles against 9.1 and later (the org.apache.lucene.tests.util package); older releases were checked with a plain main() probe.

As the regression test: the same test, for example as a new method in TestCoreParser.

cp $REPRO/TestClauseDefaultOccurs.java $LUCENE/lucene/queryparser/src/test/org/apache/lucene/queryparser/xml/
cd $LUCENE && ./gradlew :lucene:queryparser:test --tests org.apache.lucene.queryparser.xml.TestClauseDefaultOccurs -Ptests.seed=DEADBEEF
# assertions off: add -Ptests.asserts=false

$ ./gradlew :lucene:queryparser:test --tests org.apache.lucene.queryparser.xml.TestClauseDefaultOccurs -Ptests.seed=DEADBEEF

TestClauseDefaultOccurs > testClauseWithoutOccursIsShould FAILED
    java.lang.AssertionError: expected one SHOULD clause f:a (default occurs), got org.apache.lucene.queryparser.xml.ParserException: Invalid value for "occurs" attribute of clause:
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:C1CC01360CA0A832]:0)
        at org.apache.lucene.queryparser.xml.TestClauseDefaultOccurs.testClauseWithoutOccursIsShould(TestClauseDefaultOccurs.java:44)

        Caused by:
        org.apache.lucene.queryparser.xml.ParserException: Invalid value for "occurs" attribute of clause:
            at app//org.apache.lucene.queryparser.xml.builders.BooleanQueryBuilder.getOccursValue(BooleanQueryBuilder.java:81)
            at app//org.apache.lucene.queryparser.xml.builders.BooleanQueryBuilder.getQuery(BooleanQueryBuilder.java:54)
            at app//org.apache.lucene.queryparser.xml.QueryBuilderFactory.getQuery(QueryBuilderFactory.java:34)
            at app//org.apache.lucene.queryparser.xml.CoreParser.getQuery(CoreParser.java:210)
            at app//org.apache.lucene.queryparser.xml.CoreParser.parse(CoreParser.java:142)
            at app//org.apache.lucene.queryparser.xml.TestClauseDefaultOccurs.testClauseWithoutOccursIsShould(TestClauseDefaultOccurs.java:42)

BUILD FAILED in 1m 8s

• The builder's own default. BooleanQueryBuilder.java:72 returns SHOULD for occs == null. That branch maps an attribute read as missing to SHOULD, and with the JDK DOM it cannot run.
• The rewrite kept the old null case and lost the empty case. Before LUCENE-6270 the method started from SHOULD, mapped "should" or "" to SHOULD (("should".equalsIgnoreCase(occs)) || ("".equals(occs))) and threw only for an unknown value that was not null (if (occs != null)). bed131f1808 (2015-02-21) rewrote the method to add occurs="filter", one part of LUCENE-6270's replacement of TermsFilter with TermsQuery, and turned that into occs == null || "should".equalsIgnoreCase(occs).
• Documentation. LuceneCoreQuery.dtd:123-124: "Controls if the clause is optional (should), mandatory (must) or unacceptable (mustNot)" and <!ATTLIST Clause occurs (should | must | mustnot) "should">; the DTD says it "describes the XML syntax" and that "CoreParser.java is the Java class that encapsulates this parser behaviour" (:20, :66). The generated docs (lucene/queryparser/docs/xml/LuceneCoreQuery.dtd.html:136, :142) say "Default value: should". No javadoc, CHANGES.txt or MIGRATE.md entry makes occurs required.
• History. From the parser's first commit (2006; the 1.9 release has the identical file) a missing or empty occurs meant SHOULD; 4.10.4 and 5.0.0 parse the hunt's probe to f:a +f:b. The LUCENE-6270 CHANGES entry (lucene/CHANGES.txt:11095-11096) says only "Replaced TermsFilter with TermsQuery, use a QueryWrapperFilter(TermsQuery) instead."; the 5.1.0 MIGRATE.txt and the issue's 12 comments never mention occurs. No upstream test asserts the rejection.
• Precedent. LUCENE-7219 (#8274, 6.0.1 "Bug Fixes", lucene/CHANGES.txt:9296-9298) made the range builders' lowerTerm/upperTerm optional, matching the underlying queries: they had been read with getAttributeOrFail, and the fix reads them through DOMUtils.getAttribute(e, name, null), which treats "" as absent.

Strongest case for intended (the reviewers rated it moderate): CoreParser parses without validation and refuses to load the DTD, so DTD defaults have no runtime force; the DTD is stale elsewhere (it documents FilteredQuery, RangeFilter, NumericRangeQuery, BooleanFilter and disableCoord, which core no longer builds, and omits occurs="filter"); every upstream test, stylesheet and DTD example writes occurs out; and the failure is loud. So occurs is effectively required and the DTD is documentation debt.

Why it fails: the non-validating parser is exactly why the builder must apply the default itself, and it tries to: the occs == null branch reads as "missing means SHOULD", and the same builder applies the DTD's defaults for boost and minimumNumberShouldMatch. The stale DTD entries describe removed elements; Clause and occurs are live. A deliberate tightening in a minor release would have come with a CHANGES or MIGRATE note and a "missing attribute" message, not Invalid value ... : with an empty value. Fixtures that always write occurs pin nothing about the absent case. In our reading the argument limits severity, not validity; this is evidence about intent, not proof of it.

Both options were run through gradle in a scratch worktree of 645d4fe14a20 and restore the pre-5.1 behaviour: a missing or empty occurs means SHOULD; explicit values, invalid values and the error message are unchanged.

• Not recommended: declare occurs required (DTD #REQUIRED, regenerated docs, a "missing occurs" message). It would document today's behaviour but contradicts the builder's own default, the docs since 2007 and the 1.9-5.0 behaviour. Not run.
• A design choice, not run: to keep occurs="" an error (it is not one of the DTD's values), test !clauseElem.hasAttribute("occurs") instead of isEmpty(); that differs from pre-5.1 behaviour and from how DOMUtils treats empty attributes.
• CHANGES entry: say plainly that a missing occurs means SHOULD (optional), as the DTD documents, not MUST or FILTER, so nobody who builds restriction clauses from templates reads the restored default as restrictive.

In BooleanQueryBuilder.getOccursValue (BooleanQueryBuilder.java:72), add occs.isEmpty() to the SHOULD test.

   static BooleanClause.Occur getOccursValue(Element clauseElem) throws ParserException {
     String occs = clauseElem.getAttribute("occurs");
-    if (occs == null || "should".equalsIgnoreCase(occs)) {
+    if (occs == null || occs.isEmpty() || "should".equalsIgnoreCase(occs)) {
       return BooleanClause.Occur.SHOULD;

Trade-offs

• Restores the 1.9-5.0 behaviour for a missing or empty occurs and matches the DTD default; occurs="filter", added in 5.1, keeps working.
• Keeps the null guard, which is dead with the JDK DOM; option B drops it.
• An explicit occurs="" becomes SHOULD, as before 5.1 and as DOMUtils treats empty attributes.

What the checks validated

Gradle, scratch worktree of 645d4fe14a20 (2026-11-01): the repro fails on the stock tree and passes with A, with assertions on and off; the whole :lucene:queryparser:test task passes with A and the repro (534 tests in 31 classes, 0 failures, 1 skipped); :lucene:queryparser:check -x test passes. Earlier, with A shadowed over the main jars, the three upstream XML parser test classes pass 69 of 69, as without it.

Read the attribute through DOMUtils.getAttribute(clauseElem, "occurs", "should"), which maps null or "" to the default, and drop the dead null test.

   static BooleanClause.Occur getOccursValue(Element clauseElem) throws ParserException {
-    String occs = clauseElem.getAttribute("occurs");
-    if (occs == null || "should".equalsIgnoreCase(occs)) {
+    // a missing (or empty) occurs attribute means SHOULD, the default in LuceneCoreQuery.dtd
+    String occs = DOMUtils.getAttribute(clauseElem, "occurs", "should");
+    if ("should".equalsIgnoreCase(occs)) {
       return BooleanClause.Occur.SHOULD;

Trade-offs

• Same behaviour as A, written the way getQuery already reads minimumNumberShouldMatch and boost and the way LUCENE-7219 fixed the range builders.
• States the default in one place; DOMUtils is already imported.

What the checks validated

Gradle, same worktree: the repro passes with B, with assertions on and off; the org.apache.lucene.queryparser.xml test classes pass (70 tests incl. the repro, 0 failures); :lucene:queryparser:check -x test passes. The whole module was not run for B.

Method: non-formal (code reading, then executed tests). On 2026-10-26 the hunt unit for the flexible, simple and XML query parsers set up, as its secondary method, a comparison of each XML builder's output with the syntax LuceneCoreQuery.dtd documents. Reading getOccursValue against the DTD's occurs default showed the null test that the DOM never satisfies. A public-API JUnit test reproduced the rejection on main, a main() probe on release jars found the boundary (5.0.0 accepts, 5.1.0 rejects), independent checkers reproduced it with their own tests, and it was cut to the one-test repro.