MatchRegionRetriever.highlightDocument throws NullPointerException when a custom FieldValueProvider returns null for a matched field, which FieldValueProvider#getValues documents as allowed ('null if the field is not loaded or does not exist'), instead of returning no hit regions

Finding LFV-606Area: highlighter / matchhighlight: MatchRegionRetriever low-level API (highlightDocument with a custom FieldValueProvider) and the offset strategies OffsetsFromPositions, OffsetsFromTokens, OffsetsFromValues and the OffsetsFromMatchIterator fallbackPriority: lowImpact: crashNovelty: new

Finding page for LFV-606

AI-assisted finding, confirmed by automated checks, not yet triaged by a human Lucene committer. This site files nothing upstream.

What it shows

You call MatchRegionRetriever.highlightDocument with your own FieldValueProvider, for example one backed by a Map built from the document's stored fields, and for a field that is indexed but not stored it returns null, exactly as FieldValueProvider#getValues documents ('null if the field is not loaded or does not exist'). You expect no hit regions for that field, which is what the same provider gives when it returns an empty list and what the in-tree highlightDocuments path gives on the same index. Instead the default offset strategy uses the null list and the call throws NullPointerException (OffsetsFromPositions.java:70 for a positions-only field, OffsetsFromTokens.java:74 for a DOCS or DOCS_AND_FREQS field; the opt-in OffsetsFromValues fails at OffsetsFromValues.java:51).

The test shows that MatchRegionRetriever.highlightDocument throws a NullPointerException when a custom FieldValueProvider returns null for a matched, unstored positions field, which FieldValueProvider#getValues documents as allowed ("null if the field is not loaded or does not exist").

Expected vs actual

Expected: no hit regions (empty map); actual: NullPointerException at OffsetsFromPositions.java:70 ("Cannot invoke List.isEmpty() because values is null"), with -ea and -da.

Files

src/org/apache/lucene/search/matchhighlight/TestFieldValueProviderNullValues.java (JUnit test)

Run it

Tested on Lucene main e1faeba08bde (2026-11-02): still reproduces.
Tested on Lucene main 645d4fe14a20 (2026-10-28): reproduced.
Recorded outcome of run 1 on Lucene main 645d4fe14a20: 1 test(s) run, 1 failed.
How to read the outcome: a failing test is the finding reproducing (the failure message states expected against actual).
Seeds: a rerun with the same seed replays the same random choices. Assertions off (-da -dsa -Dtests.asserts=false) is the production-like setting.

# Lucene checkout at the tested commit, built with: ./gradlew --no-daemon assemble   (JDK 25)
LUCENE=/path/to/lucene; V=12.0.0-SNAPSHOT
CP=$LUCENE/lucene/core/build/libs/lucene-core-$V.jar
CP=$CP:$LUCENE/lucene/codecs/build/libs/lucene-codecs-$V.jar
CP=$CP:$LUCENE/lucene/test-framework/build/libs/lucene-test-framework-$V.jar
CP=$CP:$LUCENE/lucene/highlighter/build/libs/lucene-highlighter-$V.jar
CP=$CP:$LUCENE/lucene/memory/build/libs/lucene-memory-$V.jar
CP=$CP:$LUCENE/lucene/queries/build/libs/lucene-queries-$V.jar
CP=$CP:randomizedtesting-runner-2.9.2.jar:junit-4.13.2.jar:hamcrest-3.0.jar   # test-framework dependencies (Maven Central)
CP=$CP:junit-platform-console-standalone-6.1.3.jar   # JUnit Platform console launcher (Maven Central)

javac -d classes -cp "$CP" $(find src -name '*.java')

# run 1: assertions on, seed DEADBEEF
java -ea -esa --add-modules jdk.incubator.vector --enable-native-access=ALL-UNNAMED -Dtests.seed=DEADBEEF \
     -cp "classes:$CP" org.junit.platform.console.ConsoleLauncher \
     execute --select-class org.apache.lucene.search.matchhighlight.TestFieldValueProviderNullValues

Gradle alternative: copy the test into the matching module's src/test/ tree of the checkout and run ./gradlew -p lucene/<module> test --tests <class> -Ptests.seed=<seed> (add -Ptests.asserts=false for a run without assertions).

Feedback

Open an issue in lucene-ai-bug-hunt. Please do not open issues or pull requests in apache/lucene about this finding unless a Lucene committer agrees it should be filed.

Snapshot of 2026-11-03. AI-assisted; not yet triaged by a human Lucene committer. Feedback: open an issue in lucene-ai-bug-hunt.