MatchRegionRetriever.highlightDocument throws NullPointerException when a FieldValueProvider returns the null its javadoc allows

MatchRegionRetriever.highlightDocument is a public low-level method that "can be used with custom document FieldValueProvider" (MatchRegionRetriever.java:390-393 at 645d4fe14a20). FieldValueProvider#getValues documents that it returns the field's values "or null if the field is not loaded or does not exist for the field" (:100-104). When a custom provider returns that null for a field the query matched, the default offset strategies use it without a check: values.isEmpty() at OffsetsFromPositions.java:70 for a positions field, values.size() at OffsetsFromTokens.java:74 for a DOCS or DOCS_AND_FREQS field, and, for a field indexed with offsets, the OffsetsFromMatchIterator.java:66 fallback whenever a match carries no offsets (for example an Intervals.extend query). The public, non-default OffsetsFromValues loops over it at OffsetsFromValues.java:51. The call throws NullPointerException instead of returning no hit regions; on main, no hit regions is what an empty list gives on the same index and what the built-in highlightDocuments(...) path gives, because its private provider maps a missing field to List.of() (:535-538). The null clause came with #12881, so every release from 10.0.0 to 10.6.0 and main are affected. In the cases run: no wrong results, no corruption, no hang, one clean exception for that call.

1. The contract. FieldValueProvider is a public interface: "@return Return a list of values for the provided field name or {@code null} if the field is not loaded or does not exist for the field." (MatchRegionRetriever.java:99-105).
2. The entry point. highlightDocument(LeafReaderContext, int, FieldValueProvider doc, ToIntFunction<String>, Map<String, List<OffsetRange>>) is public, documented as usable with a custom provider, and states no non-null precondition (:390-400). For each query-affected highlighted field with matches (:410-418) it calls offsetStrategy.get(matchesIterator, doc) (:434) and stores non-empty ranges (:435-437).
3. The default strategy per field (:488-505): positions without offsets use OffsetsFromPositions (:492-493); positions with offsets use OffsetsFromMatchIterator with an OffsetsFromPositions fallback (:489-490); DOCS, DOCS_AND_FREQS and DOCS_AND_CUSTOM_FREQS use OffsetsFromTokens (:495-505).
4. No strategy checks for null:
   • OffsetsFromPositions.java:59 passes doc.getValues(field) on, and :69-72 does

// If values are not stored, there is nothing to highlight...
if (values.isEmpty()) {        // :70  NullPointerException when values == null
  return List.of();
}

• OffsetsFromTokens.java:54 reads the list and :74 calls values.size().
• OffsetsFromValues.java:47 reads it and :51 loops over it (a public strategy a caller may choose; never a default).
• OffsetsFromMatchIterator.java:64-66 falls back to OffsetsFromPositions with doc.getValues(field) as soon as one match reports a negative offset (:52-57). That happens on a field indexed with offsets too: matches of Intervals.extend always report offset -1 (ExtendedIntervalsSource.java:89-101 in lucene/queries), and Lucene's own testDegenerateIntervalsWithOffsets (TestMatchRegionRetriever.java:393-415) uses this fallback with the built-in provider.

1. Why in-tree callers never see it. The only in-tree provider, the private StoredFieldsVisitor.getValues (:535-538), returns values == null ? List.of() : values, and the only in-tree call of highlightDocument (:372) passes it. No Lucene test implements FieldValueProvider or calls highlightDocument directly.

On a field indexed with offsets (DOCS_AND_FREQS_AND_POSITIONS_AND_OFFSETS, unstored, "foo bar"; default strategy OffsetsFromMatchIterator; main, -ea and -da):

On releases, a positions field also throws for an empty list (RuntimeException: One of the offsets missing for position range, OffsetsFromPositions.java:146 on 10.6.0); main removed that with #15577, which is in no release. So on releases the null-specific failure is the DOCS / DOCS_AND_FREQS one.

• Who is hit. Callers of the public highlightDocument(...) with a hand-written FieldValueProvider that returns null for a highlighted field the query matched, when the field's offsets have to be recomputed from its text: a field indexed without offsets (positions-only, DOCS, DOCS_AND_FREQS), a field indexed with offsets when a match carries no offsets (for example an Intervals.extend query, whose matches report offset -1), or the opt-in OffsetsFromValues. In every such case run, the call throws.
• Default vs opt-in. Default strategies and settings, no special codec, no scale; opt-in on the API side. MatchHighlighter and highlightDocuments(...) use the built-in provider and are not affected. On a field indexed with offsets, a query whose matches carry offsets (a term query, for example) never reads the values. The recorded GitHub code searches (default branches) found no use of MatchRegionRetriever in Solr, Elasticsearch or OpenSearch.
• Effect. An unchecked NullPointerException for that document's highlight call. In the cases run: no wrong regions, no corruption, no hang; 1,000 failing calls left no state behind in the retriever or the shared analyzer. When several fields are highlighted, the caller's output map may already hold the (correct) ranges of fields handled before the failing one, so discard it after a failure. Whether it costs one snippet or a whole result page depends on the application's own error handling.
• Workaround. Return List.of() instead of null.

The try/catch only turns the NPE into an AssertionError that states expected vs actual. The only randomness is the test framework's per-seed default codec, locale and time zone (seed DEADBEEF ran with codec=Asserting(Lucene104), locale=brx, timezone=Asia/Tokyo); the result was the same for every seed tried (DEADBEEF, 5EED1234 three times, 7EB4A001, 7EB4A002).

On 10.0.0 to 10.6.0 the test also fails with the NPE (at OffsetsFromPositions.java:96), but there a positions field also throws for an empty list (#15577 is in no release). To show the null-specific failure on a release, use a DOCS_AND_FREQS field: an empty list gives no regions and null throws at OffsetsFromTokens.java:74. Those runs compiled the test with JDK 25 against the release jars; 10.x builds at Java 21, which rejects the _ lambda parameters, so in a 10.x checkout rename them (for example f -> false).

Suggested regression test: this same test, as its own class or as one method in TestMatchRegionRetriever.

# main 645d4fe14a20, gradle, seed DEADBEEF, -Ptests.asserts=true (the same with false)
TestFieldValueProviderNullValues > testNullValuesGiveNoHighlights FAILED
    java.lang.AssertionError: expected no highlights for null field values, got java.lang.NullPointerException: Cannot invoke "java.util.List.isEmpty()" because "values" is null
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:A4A4612B6860C9EF]:0)
        at org.apache.lucene.search.matchhighlight.TestFieldValueProviderNullValues.testNullValuesGiveNoHighlights(TestFieldValueProviderNullValues.java:73)

        Caused by:
        java.lang.NullPointerException: Cannot invoke "java.util.List.isEmpty()" because "values" is null
            at org.apache.lucene.search.matchhighlight.OffsetsFromPositions.convertPositionsToOffsets(OffsetsFromPositions.java:70)
            at org.apache.lucene.search.matchhighlight.OffsetsFromPositions.get(OffsetsFromPositions.java:59)
            at org.apache.lucene.search.matchhighlight.MatchRegionRetriever.highlightDocument(MatchRegionRetriever.java:434)
            at org.apache.lucene.search.matchhighlight.TestFieldValueProviderNullValues.testNullValuesGiveNoHighlights(TestFieldValueProviderNullValues.java:70)
:lucene:highlighter:test (FAILURE): 1 test, 1  failure

# 10.6.0 release jar, -da: the same AssertionError, caused by
#   NullPointerException: Cannot invoke "java.util.List.size()" because "values" is null
#   at OffsetsFromPositions.convertPositionsToOffsets(OffsetsFromPositions.java:96) <- get(:59) <- MatchRegionRetriever.highlightDocument(:429)
# with the null-as-empty fix on main: 1 test, 0 failures (assertions on and off)

• The javadoc allows null and names the case: "or {@code null} if the field is not loaded or does not exist for the field" (MatchRegionRetriever.java:101-102). An indexed but unstored field is the natural reading of the "does not exist" case (the javadoc does not define it further).
• The entry point invites custom providers: "Low-level method for retrieving hit ranges for a single document. This method can be used with custom document FieldValueProvider." (:390-393). No precondition, and nothing in the package is marked @lucene.experimental or @lucene.internal.
• Main defines missing values as "nothing to highlight": #15577 (548ae997d12, 2026-01-15) added "If values are not stored, there is nothing to highlight..." and an early return for an empty list (OffsetsFromPositions.java:69-72). The test expects the same for null.
• History: #12881 (755de5aae33, 2023-12-07) replaced the 9.x interface List<CharSequence> getValues(String field) throws IOException, which had no null clause and whose default provider never returned null. In the same commit it added the null clause, wrote the new provider to return List.of() for a missing field, and changed the strategies only from List<CharSequence> to List<String>. Its body, reviews and comments never discuss null; MIGRATE.md calls it "Minor API changes", and neither it nor CHANGES.txt mentions null.
• Tests: no upstream test returns null from a provider or expects an exception for it.

Strongest case (from the intended-behaviour checker, the cross-model reviewer and the adjudication): a custom provider is expected to supply every query-affected highlighted field, as the constructors' field-loading guidance describes (MatchRegionRetriever.java:118-128 and :153-163), and the FieldValueProvider javadoc itself sends readers to that guidance "on which fields are available through this interface" (:93-98); the null clause is a javadoc slip (its wording, "does not exist for the field", is garbled); the strategies date from 9.x, where null was impossible; and the method is labelled low-level. So the javadoc, not the code, should change.

Why it fails, in part: the cross-reference makes the argument stronger than it first looks, but the guidance only says which fields the predicates load; it does not require non-null values or say what null means, while the getValues javadoc explicitly allows null. And "does not exist" is not a loading question: a provider that loads every stored field and returns null only for a field the document does not store still throws, while the empty-list and built-in controls on the same index give no regions. A javadoc-only fix is still a fair outcome.

The recommended fix was applied with git apply to a scratch worktree of 645d4fe14a20 and run through gradle; the alternatives were not run.

In OffsetsFromPositions.convertPositionsToOffsets change the early return to if (values == null || values.isEmpty()) return List.of();; in OffsetsFromTokens.get and OffsetsFromValues.get return List.of() right after doc.getValues(field) when it is null.

// OffsetsFromPositions.convertPositionsToOffsets
// If values are not stored (or not loaded), there is nothing to highlight...
if (values == null || values.isEmpty()) {
  return List.of();
}

// OffsetsFromTokens.get and OffsetsFromValues.get
List<String> values = doc.getValues(field);
if (values == null) {
  return List.of();
}

Trade-offs

• Matches the javadoc and the existing "nothing to highlight" early return; non-null lists behave exactly as before, and the check is one comparison per highlighted field.
• Covers custom callers of the public strategy classes too. A custom OffsetsRetrievalStrategy that calls getValues still has to handle null itself.
• A 10.x backport needs the same checks; there a positions field also throws for an empty list (fixed on main by #15577), so the backport may want #15577's early return as well.
• Open PR #16 (2021) adds one more getValues loop in OffsetsFromMatchIterator; it would need the same check if it is ever merged.

What the checks validated

• Gradle, scratch worktree of 645d4fe14a20 with the repro test added, seed DEADBEEF: the repro test passes with -Ptests.asserts=true and false.
• The whole :lucene:highlighter:test with the fix, assertions on: 35 classes, 723 tests, 0 failures. The same suite without the fix: 723 tests, 1 failure, the repro.
• :lucene:highlighter:checkGoogleJavaFormat passes with the fix.
• Earlier hunt runs: with the fix shadowed, the minimal test (main, -ea and -da), the independent reproduction's 7-test class (7 of 7) and the intended-behaviour check's unstored-field test (5 of 5) pass; the 48 matchhighlight tests passed with it on a9779ef3c193.

Rewrite the getValues javadoc to say that a field with no values (not loaded or absent) returns an empty list, never null.

Trade-offs

No code change and no cost, and it matches what the built-in provider already does. But existing custom providers that followed the current javadoc keep failing with an NPE, and nothing tells them why; a clear exception or the null check would still help them.

What the checks validated

Not run.

Wrap the caller's provider in highlightDocument so that getValues returns List.of() for null, before any strategy sees it.

Trade-offs

One place, and it also protects custom strategies called from highlightDocument. But it does not cover callers that use the public strategy classes with their own provider outside highlightDocument, and it adds a wrapper object per document.

What the checks validated

Not run.

Method: non-formal (code reading, then a directed JUnit), 2026-10-26, main a9779ef3c193. The hunt unit built an end-to-end model of the matchhighlight passages path; that model drives only highlightDocuments(...), whose provider never returns null. Reading the interface it relies on showed the documented null return and the unguarded strategies. The finder's JUnit (a provider that returns null for every field: positions field, DOCS_AND_FREQS field, offsets control) failed 2 of 3 with -ea and -da. A later minimisation step replaced it with the one-method test shown here.