A PayloadScoreQuery that wraps another PayloadScoreQuery sees no payloads, and with the 3-argument constructor every matching search throws

PayloadScoreQuery scores span matches by the payloads stored at the matched positions, and its constructors accept any SpanQuery, including another PayloadScoreQuery. When one wraps another, directly or through a same-field FieldMaskingSpanQuery, the outer weight scores with the inner weight's SimScorer, which is null because a PayloadSpanWeight passes no term statistics to SpanWeight. SpanScorer treats a null SimScorer as "scores not required" and returns before it collects payloads.

So with includeSpanScore=false every hit gets the payload function's no-payload value (1.0 for the built-in functions) instead of its payload score, silently, and with includeSpanScore=true, the 3-argument constructor's default, every search in which that query matches a document throws (AssertionError with -ea, NullPointerException with -da). The match set is unchanged. Only Java code that builds the nested shape itself is hit; no Lucene query parser builds it. 5.3.0 (2015) to 10.6.0 and main are affected (22 releases and main run; all 95 release tags in that range have the two defect lines); a one-file fix is validated.

1. A payload weight exists only when scores are needed. PayloadScoreQuery.createWeight asks the wrapped query for its weight with the same score mode and, when scores are needed, wraps it in a PayloadSpanWeight (PayloadScoreQuery.java:118-125). A nested inner PayloadScoreQuery therefore builds a PayloadSpanWeight too.
2. That weight has no SimScorer. Its constructor calls super(PayloadScoreQuery.this, searcher, null, boost) (:150). SpanWeight returns a null SimScorer for null term states (SpanWeight.java:98, :104), getSimScorer() returns that field (:160-163), and PayloadSpanWeight does not override it. The SpanWeight javadoc allows null term states only "if scores are not required" (:88-89).
3. Its scorer borrows the inner weight's SimScorer: new PayloadSpanScorer(payloadSpans, innerWeight.getSimScorer(), norms) (PayloadScoreQuery.java:196). For a SpanTermQuery inner weight that exists; for an inner PayloadSpanWeight it is null.
4. A null SimScorer skips the payloads. SpanScorer.setFreqCurrentDoc does if (scorer == null) { // scores not required, break out here then freq = 1; return; before spans.doCurrentSpans() (SpanScorer.java:108-113), which is where PayloadSpans collects payloads (PayloadScoreQuery.java:243-245). The function then returns its no-payload value: MaxPayloadFunction.docScore is numPayloadsSeen > 0 ? payloadScore : 1 (MaxPayloadFunction.java:42-43).
5. With the span score it throws. PayloadSpanScorer.scoreCurrentDoc multiplies by super.scoreCurrentDoc() when includeSpanScore is true (PayloadScoreQuery.java:284-294), which asserts scorer != null : ... " has a null docScorer!" (SpanScorer.java:75) and then calls scorer.score(freq, norm) (:80).
6. Second trigger. FieldMaskingSpanQuery.createWeight returns the masked query's weight (FieldMaskingSpanQuery.java:88-92), so a same-field mask hands the outer query the same inner PayloadSpanWeight.
7. Why a real span query in between works. Every other wrapping span weight builds its own SimScorer with scoreMode.needsScores() ? getTermStates(...) : null (SpanPositionCheckQuery.java:70, SpanNearQuery.java:181, SpanOrQuery.java:131, SpanNotQuery.java:116, SpanContainingQuery.java:53, SpanWithinQuery.java:52, SpanPayloadCheckQuery.java:113). PayloadScoreQuery.java:196 is the only main-source line in lucene/queries that scores with another weight's SimScorer.

A plain main() without the test framework (production default codec Lucene104; payloads from WhitespaceTokenizer + DelimitedPayloadTokenFilter); two documents, f = x|3 and x|5; main 645d4fe14a20, assertions on:

explain() agrees with the wrong score ("1.0 = MaxPayloadFunction.docScore()"). The same four mismatches on 10.6.0 and 10.5.2. 1.0 is the no-payload value of all four built-in functions; a custom count function scored 0.0.

No javadoc or test defines the exact score of a nested query with includeSpanScore=true. The expected values above are what the same queries give when a real span query (SpanFirstQuery) separates them; the unambiguous defect is that the outer function collects no payloads and that matching searches throw.

• What goes wrong. With includeSpanScore=false every hit of that query gets the same score (1.0 for the built-in functions): the payload ranking is lost, with no error, and explain() reports the same 1.0. With includeSpanScore=true, the 3-argument constructor's default, every search in which that query matches a document throws (AssertionError with -ea, NullPointerException with -da); a search with no matching document does not, because the null SimScorer is only used to score a hit (and scorerSupplier returns null for a segment without spans, PayloadScoreQuery.java:190-192).
• Limits. The match set is unchanged; nothing is written to the index; no hang or resource growth was observed in any run.
• No special setup. Default IndexWriterConfig, codec and searcher; the plain-Java main() fails with the production default codec (Lucene104) and no test framework; the JUnit tests fail under whichever codec the test framework picks (Lucene104 and Asserting(Lucene104) were recorded). Runs used one- and two-document indexes; nothing in the mechanism depends on index size.
• Reach: rare. No Lucene query parser builds the shape: the XML parser's payload builder always wraps a SpanTermQuery (BoostingTermBuilder.java:38-41), and neither does Solr's payload_score parser. GitHub code search finds no code that nests the constructors. By reading only, a 2021 outside article describes a search-engine plugin whose builder wraps any parsed span query in a PayloadScoreQuery, so a nested request could build the shape there (not run).
• Context. PayloadFunction is @lucene.experimental (PayloadFunction.java:27), and open issue #16820 proposes deprecating span queries.

The token is built with the test framework's Token and CannedTokenStream, so only the queries module is needed. It fails the same way with seeds DEADBEEF, 5EED1234, 7B1E0834 and 7B1E0835, through gradle and the standalone runner, and on the 10.6.0 and 10.5.2 release jars.

The test shows the silent form. The throwing form has the same cause and the same fix: build both queries with the 3-argument constructor and search throws (AssertionError at SpanScorer.java:75 with -ea, NullPointerException at :80 with -da).

The file fails unchanged on 9.1.0, 10.5.2 and 10.6.0 (the releases it was run on); before 9.1.0 CannedTokenStream lives in another package.

The one-byte payload is valid for PayloadDecoder.FLOAT_DECODER: despite its javadoc ("interprets the bytes of a payload as a float"), it returns the payload's first byte as a number, bytes.bytes[bytes.offset], not a 4-byte float (PayloadDecoder.java:28-29), so {3} decodes to 3.0.

TestNestedPayloadScoreQuery > testNestedPayloadScore FAILED
    java.lang.AssertionError: score of the nested PayloadScoreQuery expected:<3.0> but was:<1.0>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:6F06E58C236B1799]:0)
        at junit@4.13.2/org.junit.Assert.fail(Assert.java:89)
        at junit@4.13.2/org.junit.Assert.failNotEquals(Assert.java:835)
        at junit@4.13.2/org.junit.Assert.assertEquals(Assert.java:577)
        at org.apache.lucene.queries.payloads.TestNestedPayloadScoreQuery.testNestedPayloadScore(TestNestedPayloadScoreQuery.java:57)

# the throwing form (3-argument constructors), from the hunt's four-test class on the same commit:
-ea: java.lang.AssertionError: class org.apache.lucene.queries.payloads.PayloadScoreQuery$PayloadSpanScorer has a null docScorer!
       org.apache.lucene.queries.spans.SpanScorer.scoreCurrentDoc(SpanScorer.java:75)
       org.apache.lucene.queries.payloads.PayloadScoreQuery$PayloadSpanScorer.getSpanScore(PayloadScoreQuery.java:285)
       org.apache.lucene.queries.payloads.PayloadScoreQuery$PayloadSpanScorer.scoreCurrentDoc(PayloadScoreQuery.java:291)
       org.apache.lucene.queries.spans.SpanScorer.score(SpanScorer.java:137)
       ...
-da: java.lang.NullPointerException: Cannot invoke "org.apache.lucene.search.similarities.Similarity$SimScorer.score(float, long)" because "this.scorer" is null
       org.apache.lucene.queries.spans.SpanScorer.scoreCurrentDoc(SpanScorer.java:80)
       org.apache.lucene.queries.payloads.PayloadScoreQuery$PayloadSpanScorer.getSpanScore(PayloadScoreQuery.java:285)
       org.apache.lucene.queries.payloads.PayloadScoreQuery$PayloadSpanScorer.scoreCurrentDoc(PayloadScoreQuery.java:291)
       org.apache.lucene.queries.spans.SpanScorer.score(SpanScorer.java:137)
       ...

• The API accepts it. The constructors take any non-null SpanQuery (PayloadScoreQuery.java:63-83); the class javadoc says it modifies "the score of a wrapped SpanQuery" (:43-47); the package javadoc says "For all terms matched by a SpanQuery, boost the score based on the value of the payload located at those terms" (package-info.java:24-26); CHANGES for LUCENE-6706 says "use PayloadScoreQuery to wrap any SpanQuery" and that it generalises payload scoring "to all Span queries" (CHANGES.txt:9407-9408, :10351-10352).
• SpanWeight's contract. Null term states "May be null if scores are not required" (SpanWeight.java:88-89); PayloadSpanWeight passes null although it is built only when scores are required (PayloadScoreQuery.java:121-124, :150).
• Siblings. Every other wrapping span weight builds its own SimScorer. The test sources' delegating AssertingSpanWeight also passes null term states but overrides getSimScorer() to return the wrapped weight's (AssertingSpanWeight.java:41, :58-60); that override came with LUCENE-6845 (2015), from the same author as PayloadScoreQuery, and PayloadSpanWeight never got one.
• One step away it works. With SpanFirstQuery (or an all-positions SpanPositionRangeQuery) between the two, the outer function sees the payloads and the scores match the non-nested query.
• Why tests missed it. No upstream test nests PayloadScoreQuery or wraps a FieldMaskingSpanQuery in one; neither did the introducing commit's tests.

The strongest case for "intended": span queries score only at the top level (LUCENE-8143 removed SpanBoostQuery as "a no-op unless used at the top level of a SpanQuery nested structure", MIGRATE.md:1604-1607), so nesting a payload query is redundant; no javadoc or test says nesting must work; PayloadFunction is experimental; FieldMaskingSpanQuery warns that masking "may lead to exceptions, poor performance, and unexpected scoring behavior" (FieldMaskingSpanQuery.java:63-65); and #16820 proposes deprecating span queries.

Why it fails: the top-level rule explains the inner function being ignored, but here the top-level query loses its own function or every search that matches a document throws; a query the constructors accept should not make IndexSearcher.search throw NullPointerException; the masking warning is about a different field's statistics, while the trigger masks the same field and direct nesting masks nothing; a null SimScorer means "scores not required" in both SpanWeight and SpanScorer, and this weight exists only when scores are required. Verdict: not intended, a defect (finder, an execution-capable cross-model review, the independent reproduction, the intended-behaviour check, the adjudication, the reproducer and the tie-break).

Three fixes were run on the 645d4fe14a20 jars; the recommended one and the hunt's were also run through gradle in a scratch worktree of 645d4fe14a20. A probe for this page (four nested shapes, not the repro) checked that nested scores equal the non-nested ones and that explain() agrees with them under CheckHits.checkExplanations(..., deep=true).

In PayloadScoreQuery.PayloadSpanWeight, add the override below, and in explain replace innerWeight.explain(context, doc) (PayloadScoreQuery.java:179-180) with explainSpanScore(context, doc). The override is the same idea as AssertingSpanWeight.getSimScorer(); the explain change keeps the "product of" details equal to the score for nested queries. The fix gives a nested query the scores the same queries get when a real span query separates them; the exact nested score with includeSpanScore=true is undocumented, so this is one reasonable choice for the maintainers.

    @Override
    public SimScorer getSimScorer() {
      return innerWeight.getSimScorer();
    }

    /** Explains the span score, which comes from the innermost non-payload span weight. */
    private Explanation explainSpanScore(LeafReaderContext context, int doc) throws IOException {
      if (innerWeight instanceof PayloadSpanWeight inner) {
        return inner.explainSpanScore(context, doc);
      }
      return innerWeight.explain(context, doc);
    }

Trade-offs

• In-tree queries that work today take the same code paths, so their scores and explanations are unchanged (executed for the non-nested cross-field FieldMaskingSpanQuery form: 0.2635535/1.1898826 before and after). SpanWeight.getSimScorer() is public, so code outside Lucene that calls it on a PayloadSpanWeight would now get the inner SimScorer instead of null (not checked).
• The inner query's payload function still contributes nothing to a nested score; the outer function now sees the inner spans' payloads. A maintainer may prefer to forbid nesting instead.
• Open PR #15819 rewrites the scorerSupplier line (:196), which this fix leaves alone; the two apply together without a textual conflict, in either order. But #15819 takes the span score from the inner weight's new createSpanScorer(), which builds the scorer from the simScorer field, null for an inner PayloadSpanWeight, so whichever change lands second must also cover that path (read, not built). The hunt's fix rewrites :196 and does conflict with #15819.

What the checks validated

• Gradle, scratch worktree of 645d4fe14a20, seed DEADBEEF: the repro fails on stock and passes with the fix, assertions on and off.
• org.apache.lucene.queries.payloads.* and org.apache.lucene.queries.spans.* with nightly tests: 267 tests, 2 skipped, 0 failures.
• The whole :lucene:queries:test: 513 tests, 5 skipped, 0 failures.
• :lucene:queries:check -x test (google-java-format, ecj lint, forbidden APIs) passes with the repro test in place.
• The explain probe passes 4 of 4. Shadowed into the 645d4fe14a20 jars, the hunt's four tests, the tie-break's JUnit (25 runs) and its plain main() pass, assertions on and off.

Pass getTermStates(innerWeight) to super, score with this weight's own getSimScorer(), and use super.explain(context, doc) for the span part.

Trade-offs

Matches how every other wrapping span weight builds its SimScorer, but changes the scores of an existing non-nested query: PSQ(FieldMaskingSpanQuery(SpanTermQuery(g:x), "f"), Max, true) goes from 0.2635535/1.1898826 to 0.57384163/1.5741748, because the field statistics move from g to f (which is what FieldMaskingSpanQuery's javadoc describes; the term statistics stay those of g:x). Shown for that one composition only. It fits only where that score change is wanted.

What the checks validated

• Gradle, the same scratch worktree: the repro passes; payloads.* and spans.* with nightly tests: 267 tests, 2 skipped, 0 failures; the explain probe passes 4 of 4.
• Earlier: the hunt's, the independent reproduction's and the tie-break's tests pass with it, and the six upstream payload suites pass 34 of 34 with nightly tests.

The first five lines above. Every score and every exception is fixed, but explain() for the nested includeSpanScore=true forms then has details whose product is not the score (3 of 4 probe tests fail, for example 0.2486203 x 3.0 against a score of 0.2486203); the top-level explanation value is right.

Trade-offs

Smallest change; leaves an explanation that Lucene's own CheckHits rejects for nested queries.

Throw IllegalArgumentException for a nested payload query. Not built.

Trade-offs

Turns the silent wrong score into a clear error, but cannot see through a FieldMaskingSpanQuery without unwrapping it, breaks code that nests today, and contradicts "wrap any SpanQuery".

Method: semi-formal (a hunt unit, payload and span queries against a reference semantics).

• The harness built small indexes with payloads (int, float and string payloads; one and three segments, with and without deletions) and compared Lucene's scores for each query shape with a reference implementation of payload scoring.
• As one of its checks it ran every shape also as PayloadScoreQuery(PayloadScoreQuery(shape, Sum, decoder, false), fn, decoder, incl), with the reference "same scores as PayloadScoreQuery(shape, fn, decoder, incl)", because a payload query's spans are its wrapped query's spans. So nesting was a deliberate check, not a blind discovery.
• Out of 76,177,080 payload-score checks, 12,120,610 nested ones returned 1.0 and 3,072 nested searches threw, while the non-nested and SpanPositionRangeQuery-wrapped forms matched the reference exactly.
• The finder, an AI tool, then read the root cause in the code and wrote a public-API JUnit; the one-test repro was cut from it on 2026-10-29.