SpanPayloadCheckQuery.toString() and explain() throw NullPointerException for a null payload entry, which the search itself accepts as "no payload here"

SpanPayloadCheckQuery keeps only the spans whose positions carry the expected payloads, given as a List<BytesRef>. Its matcher gives a null entry a defined meaning, "this position has no payload" (matches = payload == null, SpanPayloadCheckQuery.java:221-225 at 645d4fe14a20), while a non-null entry never matches a payload-less position (:226-230). So [null, "p"] over the span a b finds a document where a has no payload and b has p. But toString(String) passes every entry, the null too, to Term.toString(BytesRef) (:252-255), which reads termText.bytes (Term.java:116) and throws NullPointerException. IndexSearcher.explain() throws the same way for every matching document, because SpanWeight.explain puts the query string into its description (SpanWeight.java:182-183). Search, count, equals and hashCode work, and nothing reaches the index. The null meaning came with LUCENE-6716 (2015), so every release from 5.4.0 to 10.6.0 and main are affected; a one-line null-safe toString fixes it.

1. The constructors store the list unchecked. Javadoc: "@param payloadToMatch The {@link java.util.List} of payloads to match" (SpanPayloadCheckQuery.java:78, :86); this.payloadToMatch = payloadToMatch; (:97). Nothing mentions null entries.
2. The matcher reads null as "no payload here" (PayloadChecker.collectLeaf, :212-233):

BytesRef payload = postings.getPayload();
if (payloadToMatch.get(upto) == null) {   // :221
  matches = payload == null;              // :222  null entry = no payload at this position
  upto++;
  return;
}
if (payload == null) {                    // :226  a non-null entry never matches a payload-less position
  matches = false;

1. toString passes every entry to Term.toString (:246-260):

for (BytesRef bytes : payloadToMatch) {   // :252
  buffer.append(Term.toString(bytes));    // :253  NullPointerException when bytes == null

1. Term.toString(BytesRef) dereferences it: decoder.decode(ByteBuffer.wrap(termText.bytes, termText.offset, termText.length)) (Term.java:116-117, reported at :116). Query.toString() calls toString("") (Query.java:55-57).
2. Explain prints the query. IndexSearcher.explain(Query, int) (IndexSearcher.java:941-944) reaches SpanWeight.explain, whose description is "weight(" + getQuery() + " in " + doc + ... (SpanWeight.java:180-189; the no-score branch at :192-193 formats getQuery() too), so in an index with positions explain throws for every document that matches; a non-matching document gets Explanation.noMatch (SpanWeight.java:198), which does not print the query.
3. Not affected: equals and hashCode (SpanPayloadCheckQuery.java:264-280) use List.equals and Objects.hashCode, which accept null; in an index with positions, search and count do not print the query. One error path does: on a field indexed without positions, SpanPayloadCheckWeight.scorerSupplier builds an IllegalStateException message that contains the query (SpanPayloadCheckQuery.java:179-186), so on such a field the query throws this NPE instead of that IllegalStateException. It fails there either way; only the exception changes (read, not run).
4. Test framework only: AssertingIndexSearcher.rewrite calls QueryUtils.check (AssertingIndexSearcher.java:64-68), whose checkUnequal builds q1 + " equal to " + q2 eagerly (QueryUtils.java:110), so under LuceneTestCase.newSearcher even search fails (read, not run separately). The repro uses no searcher at all.

• Who is hit. Code that builds SpanPayloadCheckQuery with a null entry in payloadToMatch and then logs, prints or explains it: a log line, a slow-query log, a debug print, an explain request, or any code that puts the query into an exception message. Every such call throws, on every run.
• Default vs opt-in. Default IndexWriterConfig, codec and a plain IndexSearcher; assertions on and off. Opt-in on the API side: a lucene-queries class that applications build directly in Java code; no Lucene query parser builds it (outside tests only SpanPayloadCheckQuery.java and PayloadMatcherFactory.java name it). Solr's payload_check parser adds only encoded, non-empty payloads, never null. List.of(...) rejects null, so a caller uses Arrays.asList, Collections.singletonList or an ArrayList.
• Effect. One unchecked NullPointerException from toString() or explain(). Search and count results are correct (in an index with positions; see the mechanism, item 6); no index change, no hang, no resource growth.
• Likelihood. Rare: no public report found in the documented searches (see Prior reports) in the nearly eleven years since 5.4.0 (tagged 2015-12-10).
• Workaround. None that keeps the meaning (an empty BytesRef cannot stand in for null, see below); callers can only avoid printing or explaining such a query.

The try/catch only turns the NPE into an AssertionError that states expected vs actual. The result does not depend on the seed (DEADBEEF and 5EED1234 tried). explain() throws through the same toString() call, so a second test would show no new cause; the finder's three-test JUnit (index, search, explain) shows that path.

Suggested regression test: this same test, as its own class or as one method in TestPayloadCheckQuery. It checks only that toString() no longer throws; it does not pin the marker text. Once a maintainer picks the marker, an assertEquals on the string can be added.

# main 645d4fe14a20, gradle, seed DEADBEEF, -Ptests.asserts=true (the same with false)
TestSpanPayloadCheckQueryNullEntryToString > testToStringWithNullEntry FAILED
    java.lang.AssertionError: expected toString() to describe the query, got java.lang.NullPointerException: Cannot read field "bytes" because "termText" is null
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:B235680A74F07667]:0)
        at org.apache.lucene.queries.payloads.TestSpanPayloadCheckQueryNullEntryToString.testToStringWithNullEntry(TestSpanPayloadCheckQueryNullEntryToString.java:34)

        Caused by:
        java.lang.NullPointerException: Cannot read field "bytes" because "termText" is null
            at org.apache.lucene.core@11.0.0-SNAPSHOT/org.apache.lucene.index.Term.toString(Term.java:116)
            at org.apache.lucene.queries.payloads.SpanPayloadCheckQuery.toString(SpanPayloadCheckQuery.java:253)
            at org.apache.lucene.core@11.0.0-SNAPSHOT/org.apache.lucene.search.Query.toString(Query.java:56)
            at org.apache.lucene.queries.payloads.TestSpanPayloadCheckQueryNullEntryToString.testToStringWithNullEntry(TestSpanPayloadCheckQueryNullEntryToString.java:31)

1 test completed, 1 failed

# 10.6.0 release jars, -ea and -da: the same NPE at Term.toString(Term.java:116) <- SpanPayloadCheckQuery.toString(SpanPayloadCheckQuery.java:254) (10.6.0 has one more line in rewrite())

# the explain path (finder's three-test JUnit, main 645d4fe14a20, -da; testSearchMatches passes):
java.lang.NullPointerException: Cannot read field "bytes" because "termText" is null
   org.apache.lucene.index.Term.toString(Term.java:116)
   org.apache.lucene.queries.payloads.SpanPayloadCheckQuery.toString(SpanPayloadCheckQuery.java:253)
   org.apache.lucene.search.Query.toString(Query.java:56)
   java.base/java.lang.String.valueOf(String.java:4530)
   org.apache.lucene.queries.spans.SpanWeight.explain(SpanWeight.java:183)
   org.apache.lucene.search.IndexSearcher.explain(IndexSearcher.java:967)
   org.apache.lucene.search.IndexSearcher.explain(IndexSearcher.java:943)

• The null meaning is deliberate code. LUCENE-6716 (22cc3a1560e, Alan Woodward (romseygeek), 2015-09-14) replaced the old collector, which skipped positions without payloads, with PayloadChecker and its explicit null branch. It survived every later change, LUCENE-9659 (2021) included.
• It is the only way to say "no payload here". The indexing chain never stores a zero-length payload (FreqProxTermsWriterPerField.java:89: if (payload != null && payload.length > 0)), and a non-null entry against a payload-less position always fails (SpanPayloadCheckQuery.java:226-230). With the stock WhitespaceTokenizer + DelimitedPayloadTokenFilter (a token without a delimiter gets setPayload(null), DelimitedPayloadTokenFilter.java:64-65), an empty-BytesRef entry matches 0 documents where a null entry matches 1.
• No precondition. Query.toString(String) "Prints a query to a string" (Query.java:47-51); IndexSearcher.explain states nothing that excludes a query it executes correctly.
• Nothing shows that the toString line considered it. It came 18 minutes after the null branch, from the same author, in "LUCENE-6716: Better toString() for SpanPayloadCheckQuery" (13a64196d0d); the first List version (bytes.utf8ToString()) also threw on null, and the available history and tests show no handling of a null entry in toString.
• Precedent. Upstream fixed toString and explain NPEs of other classes as bugs, e.g. LUCENE-8130 (#9178, TermStates.toString) and LUCENE-9524 (an NPE in SpanWeight#explain with another cause).
• Tests and docs. No upstream test builds a null entry; CHANGES.txt (the LUCENE-6716 entry) and MIGRATE.md say nothing about it.

Strongest case (the intended-behaviour checker, the cross-model reviewers and the adjudication): nothing documents null entries, toString has never been null-safe in any version since the class arrived in 3.1, and nothing in Lucene or Solr produces a null entry, so null was never meant as input and the fix is to reject it.

Why it fails: the matcher implements null on purpose, and it is the only spelling of a case the stock DelimitedPayloadTokenFilter produces all the time (tokens without a payload); the constructor accepts it and the search honours it. What the argument does change is the framing: the meaning is defined by the code, not the javadoc, so rejecting and documenting null (option B) is a fair outcome too.

The recommended fix was applied to a scratch worktree of 645d4fe14a20 and run through gradle; the alternatives were not run.

In SpanPayloadCheckQuery.toString(String) append "null" for a null entry; in the javadoc of both constructors say that a null entry matches a position that has no payload.

// SpanPayloadCheckQuery.toString(String)
for (BytesRef bytes : payloadToMatch) {
  buffer.append(bytes == null ? "null" : Term.toString(bytes));
  buffer.append(';');
}
// both constructors
 * @param payloadToMatch The {@link java.util.List} of payloads to match; a {@code null} entry
 *     matches a position that has no payload

Trade-offs

• Keeps the only way to require a position without a payload and makes it a documented contract; non-null entries print exactly as before.
• The marker text is a choice (null, or something like <none>).

What the checks validated

• Gradle, scratch worktree of 645d4fe14a20 with the repro test added, seed DEADBEEF: the repro passes with -Ptests.asserts=true and false.
• The whole :lucene:queries:test with the fix, assertions on: 52 classes (51 upstream + the repro), 513 tests, 0 failures, 5 skipped. The same suite without the fix: 1 failure, the repro.
• :lucene:queries:checkGoogleJavaFormat passes with the fix.
• With the fixed class shadowed, the finder's three-test JUnit passes 3 of 3: toString(), explain() (a matching Explanation) and the search control.

Check each entry with Objects.requireNonNull or throw IllegalArgumentException, and say so in the javadoc.

Trade-offs

• Makes the contract explicit and fails early.
• Removes the only way to match a position without a payload, and turns queries that search correctly today into construction errors.

Return "null" from Term.toString(BytesRef) for a null argument.

Trade-offs

• One place for every caller.
• Changes a core utility for a queries-module bug and could hide genuine null terms elsewhere.

Method: semi-formal (bounded-exhaustive translation validation), 2026-10-28, main a9779ef3c193. The hunt unit wrote a reference semantics of the payload span queries, independent of the matchers (a null list entry requires "no payload"), built test indexes over the terms a, b, c with and without payloads, and compared the real classes on the stock jars with the reference: all 133,080 SpanPayloadCheckQuery searches agreed, and explain() threw in 932 of the 3,380 sampled score/explain checks (every recorded example: NullPointerException on a list with a null entry). Reading the code then gave the root cause in toString(). The finder's JUnit (one document, [null, "p"], plain IndexSearcher) failed 2 of 3 with -ea and -da, the search control passing; a later minimisation step replaced it with the one-method test shown here.