ValueSource#asDoubleValuesSource and #asLongValuesSource never call createWeight, so a ValueSource that keeps per-query state throws NullPointerException

ValueSource#asDoubleValuesSource() and #asLongValuesSource() wrap a ValueSource so it can be used where Lucene expects a DoubleValuesSource or LongValuesSource, for example in FunctionScoreQuery.boostByValue, the documented replacement for the removed BoostedQuery, or in a sort. A ValueSource is used in two steps: createWeight(context, searcher) once per query, where it "can optionally store weight info in the context", then getValues(context, segment), which gets "the context that was previously passed to createWeight()". The wrappers build a fresh context for every segment and call getValues without ever calling createWeight, although their rewrite(searcher) receives the searcher that call needs. TotalTermFreqValueSource (ttf) and SumTotalTermFreqValueSource (sttf) compute their number in createWeight and return context.get(this) from getValues, so through the wrappers they return null and the search throws NullPointerException; a custom ValueSource written the way the javadoc describes finds its state missing too. One document is enough, with a default IndexWriterConfig and a plain IndexSearcher, assertions on or off. No version of the bridges has called createWeight since they appeared in 6.4.0 (2017), and every release run, 6.4.0 to 10.6.0, fails, as does main; the boostByValue form used here exists from 7.3.0, and older releases fail through a sort or new FunctionScoreQuery.

1. asDoubleValuesSource() returns new WrappedDoubleValuesSource(this, null) (ValueSource.java:183-186).
2. FunctionScoreQuery.createWeight rewrites its source with the searcher (FunctionScoreQuery.java:121); boostByValue wraps the source in MultiplicativeBoostValuesSource (:84-86), whose rewrite rewrites the bridge (:315-317). The bridge's rewrite only returns a new wrapper that holds the searcher (ValueSource.java:243-246). No createWeight.
3. Per segment, FunctionScoreWeight.scorerSupplier calls getValues (FunctionScoreQuery.java:239). The bridge builds newContext(searcher), a fresh map that holds only "searcher" (ValueSource.java:200), adds "scorer" (:203) and calls in.getValues(context, ctx) (:205), with no in.createWeight(context, searcher) before it.
4. TotalTermFreqValueSource.getValues is return (FunctionValues) context.get(this); (TotalTermFreqValueSource.java:59-62); only its createWeight (:65-81) sums the term's frequency over all segments and stores that entry. So the bridge's fv is null. SumTotalTermFreqValueSource is the same (:51-54, :57-74).
5. Scoring the document goes through FunctionScoreQuery.java:244-245 and :299 and DoubleValues.java:79 to fv.doubleVal(scorer.docId) (ValueSource.java:210): NullPointerException.
6. asLongValuesSource() has the same shape (:126-133; the NPE comes from fv.exists(doc), :144), and so has the bridge's explain (:235-240).

The native consumers do it right: FunctionQuery.FunctionWeight (FunctionQuery.java:70-71), FunctionRangeQuery (FunctionRangeQuery.java:148-149) and ValueSourceSortField.rewrite (ValueSource.java:379-382) each build one context and call createWeight(context, searcher) before any getValues.

• Minimal repro, one document "test test": expected score 2; actual NullPointerException ("this.val$fv" is null). On the same index new FunctionQuery(ttf) scores 2.0 and boostByValue over IndexReaderFunctions.totalTermFreq scores 2.0.
• Two documents ("test test x", "test y"): ttf is 3 and sttf is 5 through FunctionQuery; through the bridge, FunctionScoreQuery with ttf, boostByValue with sttf and a sort on ttf.asLongValuesSource().getSortField(false) all throw.
• A custom source that stores numDocs in createWeight: 2 through FunctionQuery; through FunctionScoreQuery and through an asLongValuesSource() sort its getValues sees a context its createWeight never filled (0 calls).
• ProductFloatFunction(const 2, ttf): 6 through FunctionQuery (on the devil's advocate's 3-document index, where ttf is 3), NullPointerException through boostByValue; FunctionQuery makes 1 createWeight call, the bridge 0. searcher.explain through the bridge throws too.

• Stateless sources (fields, constants, arithmetic on them), the score, and statistics that need only the searcher or the top-level reader work through the bridge: maxdoc() and docfreq() read the "searcher" entry that newContext adds, and numdocs() reads the top-level reader.
• QueryValueSource and ScaleFloatFunction do not crash, because they compute their state lazily when it is missing; that they redo the work per segment through the bridge is code reading only.
• A custom source that silently falls back to a default when its state is missing would give wrong scores instead of an exception; that variant was not tested.

• Who is hit. Code that passes such a source through asDoubleValuesSource() into FunctionScoreQuery (scoring, boostByValue, explain), or through asLongValuesSource() (or asDoubleValuesSource()) into a sort. In-tree: TotalTermFreqValueSource (ttf), SumTotalTermFreqValueSource (sttf) and any function containing them. Outside the tree: any custom ValueSource that stores state in createWeight, as its javadoc describes.
• On the documented migration path. MIGRATE.md:1377-1381 says BoostedQuery "may be replaced by calls to FunctionScoreQuery.boostByValue()". BoostedQuery called createWeight on its ValueSource; boostByValue reaches a ValueSource only through asDoubleValuesSource().
• Default vs opt-in. A default new IndexWriterConfig() and a plain IndexSearcher; the trigger is the opt-in combination of a stateful source with the bridge. Under Lucene's test framework the codec is randomized (the gradle runs used Asserting(Lucene104)); the codec plays no part, because ttf's term lookup lives in the createWeight that is never called, so the bridge throws before ttf reads anything.
• Scale. One document is enough. At any index size a search throws as soon as it evaluates the source for a matching document, with assertions on and off; a query that matches nothing in a segment never reaches the bridge there (FunctionScoreQuery.java:236-237; explain of a non-match returns at :177-178). The failure is loud, only that query fails, and nothing is written to the index.
• Downstream (read, not run). Apache Solr builds ttf() and sttf() from these classes, and its boost parsers wrap the boost function with asDoubleValuesSource() through boostByValue. That end-to-end path was not run.
• How likely. Rare: ttf and sttf are per-term constants, seldom useful as a per-document boost or sort key, custom stateful sources are uncommon, and the bug has gone unreported since 6.4.0 (2017).

The test indexes one document with TextField("text", "test test") in a ByteBuffersDirectory, wraps new TotalTermFreqValueSource("text", "test", "text", new BytesRef("test")) with asDoubleValuesSource(), and runs new IndexSearcher(reader).search(FunctionScoreQuery.boostByValue(new MatchAllDocsQuery(), ...), 1). Deterministic, single thread; the test framework's randomized codec plays no part (the bridge throws before ttf reads anything). TotalTermFreqValueSource is a public class but is marked @lucene.internal, so it is not a supported public API; the custom-source variant (a ValueSource that stores its state in createWeight, executed by the finder and the independent reproducer) uses only supported extension points and can replace or accompany this test in an issue. The NullPointerException is caught only to restate expected versus actual in the message; the cause is kept. It imports org.apache.lucene.tests.*, so it compiles on 9.1.0 and later; the study ran raw-Map ports of its own tests on 6.4.0 to 9.0.0. Through gradle, checkGoogleJavaFormat asks to wrap lines 45 and 46 (101 and 103 characters); ./gradlew tidy does it.

TestValueSourceBridgeTotalTermFreq > testBoostByTotalTermFreq FAILED
    java.lang.AssertionError: expected score 2 (ttf, as FunctionQuery gives), got java.lang.NullPointerException: Cannot invoke "org.apache.lucene.queries.function.FunctionValues.doubleVal(int)" because "this.val$fv" is null
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:B63616D72850F488]:0)
        at org.apache.lucene.queries.function.TestValueSourceBridgeTotalTermFreq.testBoostByTotalTermFreq(TestValueSourceBridgeTotalTermFreq.java:51)

        Caused by:
        java.lang.NullPointerException: Cannot invoke "org.apache.lucene.queries.function.FunctionValues.doubleVal(int)" because "this.val$fv" is null
            at org.apache.lucene.queries.function.ValueSource$WrappedDoubleValuesSource$1.doubleValue(ValueSource.java:210)
            at org.apache.lucene.core@11.0.0-SNAPSHOT/org.apache.lucene.search.DoubleValues$1.doubleValue(DoubleValues.java:79)
            at org.apache.lucene.queries.function.FunctionScoreQuery$MultiplicativeBoostValuesSource$1.doubleValue(FunctionScoreQuery.java:299)
            at org.apache.lucene.queries.function.FunctionScoreQuery$FunctionScoreWeight$1.score(FunctionScoreQuery.java:245)
            ...
            at org.apache.lucene.queries.function.TestValueSourceBridgeTotalTermFreq.testBoostByTotalTermFreq(TestValueSourceBridgeTotalTermFreq.java:49)

# the same with -Ptests.asserts=false; with the recommended fix the test passes

• The contract. getValues "Gets the values for this reader and the context that was previously passed to createWeight()" (ValueSource.java:43-47); createWeight: sources "can optionally store weight info in the context. The context object will be passed to getValues() where this info can be retrieved" (:65-69). The bridge is itself a caller of in.getValues and breaks that precondition; its javadoc promises to "Expose this ValueSource" (:111, :183) with no restriction.
• The bridge has what it needs. DoubleValuesSource's class javadoc tells clients to call rewrite(IndexSearcher) against the top-level searcher before getValues (DoubleValuesSource.java:34-36); FunctionScoreQuery does, and the bridge keeps the searcher. Core's DoubleValuesSource.fromQuery does its per-query setup in rewrite, building its Weight there (:776-780).
• Every other consumer calls it: FunctionQuery.java:71, FunctionRangeQuery.java:149, ValueSource.java:381, and the removed BoostedQuery (boostVal.createWeight(fcontext,searcher);, 7.2.0 BoostedQuery.java:74). LUCENE-8099 removed BoostedQuery and MIGRATE.md:1377-1381 maps it to boostByValue(), which reaches a ValueSource only through asDoubleValuesSource(); none of the 35 comments on #9147 discusses losing the call.
• Tests. Upstream tests call createWeight before getValues (TestValueSources.java:697-701, :935-940) and use the bridge only with sources that need no createWeight; no test asserts that the bridge skips it.
• History. The bridges were born without the call (LUCENE-7610, 2017), and no later change says it is skipped on purpose: not LUCENE-7736 (which gave the bridge the searcher), LUCENE-8726, LUCENE-10252 or #15406. In LUCENE-5335 (#6399, 2013) Yonik Seeley gave state keyed by this as an intended use of the context: 'SumTotalTermFreqValueSource is one example of where "this" is very conveniently used as a key'.

The strongest case for "works as designed": the DoubleValuesSource protocol has no createWeight step; ttf and sttf are @lucene.internal, and LUCENE-7736 (#8787) added IndexReaderFunctions.totalTermFreq and sumTotalTermFreq so that such statistics work in FunctionScoreQuery; the facet code the bridge replaced also skipped createWeight; QueryValueSource and ScaleFloatFunction cope with a skipped createWeight, so robust sources should too; the bridge's javadoc is one line; and open issue #16820 proposes deprecating ValueSource in Lucene 11.

Why it fails: the replacement argument covers ttf and sttf only, not a custom source that follows the createWeight javadoc and fails through public API alone; nothing limits the bridges to sources that ignore createWeight, while the getValues javadoc makes it a precondition; LUCENE-7736 put the searcher into the bridge's context, so sources that read it (such as maxdoc()) work through the bridge, and it neither added nor ruled out createWeight-dependent sources (reading it as a sign that the bridge is meant to serve them is an inference); every native consumer and the removed BoostedQuery call it; two lazy fallbacks are defensive code, not a relaxed contract; and #16820 is a proposal, while 6.4.0 to 10.6.0 are affected. No reviewer accepted the intended case: the cross-model reviews rated it weak, the devil's advocate is highly confident it is not intended, and the independent reproducer and the adjudication found the call promised by the javadoc. Verdict: not intended, a defect. Expect "use IndexReaderFunctions" and "ValueSource is being deprecated" as replies; the custom-source case and the BoostedQuery migration answer both.

Option B was applied in a scratch worktree of 645d4fe14a20 and run with gradle (the worktree was removed after the runs). Option A, the study's sketch, was validated with a standalone runner by shadowing the patched class. Options C and D were not built.

In both wrappers, rewrite(searcher) builds newContext(searcher), calls in.createWeight(context, searcher) and keeps the filled map; getValues and explain start from new IdentityHashMap<>(weightContext) instead of newContext(searcher). An un-rewritten wrapper keeps a newContext(null) map, whose only entry is "searcher" -> null, the same as today's per-segment context. WrappedLongValuesSource's now unused searcher field is replaced; WrappedDoubleValuesSource keeps its package-private searcher field, which TestValueSources checks is null before rewrite.

diff --git a/lucene/queries/src/java/org/apache/lucene/queries/function/ValueSource.java b/lucene/queries/src/java/org/apache/lucene/queries/function/ValueSource.java
index 8f31c9af02c..d7f3dcc547f 100644
--- a/lucene/queries/src/java/org/apache/lucene/queries/function/ValueSource.java
+++ b/lucene/queries/src/java/org/apache/lucene/queries/function/ValueSource.java
@@ -110,22 +110,23 @@ public abstract class ValueSource {
 
   /** Expose this ValueSource as a LongValuesSource */
   public LongValuesSource asLongValuesSource() {
-    return new WrappedLongValuesSource(this, null);
+    return new WrappedLongValuesSource(this, newContext(null));
   }
 
   private static class WrappedLongValuesSource extends LongValuesSource {
 
     private final ValueSource in;
-    private final IndexSearcher searcher;
+    // filled once per query by in.createWeight() in rewrite(), then only copied
+    private final Map<Object, Object> weightContext;
 
-    private WrappedLongValuesSource(ValueSource in, IndexSearcher searcher) {
+    private WrappedLongValuesSource(ValueSource in, Map<Object, Object> weightContext) {
       this.in = in;
-      this.searcher = searcher;
+      this.weightContext = weightContext;
     }
 
     @Override
     public LongValues getValues(LeafReaderContext ctx, DoubleValues scores) throws IOException {
-      Map<Object, Object> context = newContext(searcher);
+      Map<Object, Object> context = new IdentityHashMap<>(weightContext);
 
       var scorer = new ScorableView(scores);
       context.put("scorer", scorer);
@@ -176,28 +177,34 @@ public abstract class ValueSource {
 
     @Override
     public LongValuesSource rewrite(IndexSearcher searcher) throws IOException {
-      return new WrappedLongValuesSource(in, searcher);
+      Map<Object, Object> context = newContext(searcher);
+      in.createWeight(context, searcher);
+      return new WrappedLongValuesSource(in, context);
     }
   }
 
   /** Expose this ValueSource as a DoubleValuesSource */
   public DoubleValuesSource asDoubleValuesSource() {
-    return new WrappedDoubleValuesSource(this, null);
+    return new WrappedDoubleValuesSource(this, null, newContext(null));
   }
 
   static class WrappedDoubleValuesSource extends DoubleValuesSource {
 
     final ValueSource in;
     final IndexSearcher searcher;
+    // filled once per query by in.createWeight() in rewrite(), then only copied
+    private final Map<Object, Object> weightContext;
 
-    private WrappedDoubleValuesSource(ValueSource in, IndexSearcher searcher) {
+    private WrappedDoubleValuesSource(
+        ValueSource in, IndexSearcher searcher, Map<Object, Object> weightContext) {
       this.in = in;
       this.searcher = searcher;
+      this.weightContext = weightContext;
     }
 
     @Override
     public DoubleValues getValues(LeafReaderContext ctx, DoubleValues scores) throws IOException {
-      Map<Object, Object> context = newContext(searcher);
+      Map<Object, Object> context = new IdentityHashMap<>(weightContext);
 
       var scorer = new ScorableView(scores);
       context.put("scorer", scorer);
@@ -234,7 +241,7 @@ public abstract class ValueSource {
     @Override
     public Explanation explain(LeafReaderContext ctx, int docId, Explanation scoreExplanation)
         throws IOException {
-      Map<Object, Object> context = newContext(searcher);
+      Map<Object, Object> context = new IdentityHashMap<>(weightContext);
       context.put("scorer", new ScorableView(docId, scoreExplanation.getValue().floatValue()));
       FunctionValues fv = in.getValues(context, ctx);
       return fv.explain(docId);
@@ -242,7 +249,9 @@ public abstract class ValueSource {
 
     @Override
     public DoubleValuesSource rewrite(IndexSearcher searcher) throws IOException {
-      return new WrappedDoubleValuesSource(in, searcher);
+      Map<Object, Object> context = newContext(searcher);
+      in.createWeight(context, searcher);
+      return new WrappedDoubleValuesSource(in, searcher, context);
     }
 
     @Override

Trade-offs

• One createWeight per query, like FunctionQuery. Each segment gets its own copy of the map, so getValues may add entries ("scorer", lazily built state) without writing to a shared map: the bridge itself writes the shared map only during rewrite, which avoids the one-context-shared-by-concurrent-slices problem of the native consumers for entries added in getValues. The copy is shallow: objects that createWeight stores (for example QueryValueSource's Weight) are shared across segments, as in FunctionQuery, and a source that keeps a reference to the createWeight map and writes to it later still shares it. Whether the context must be read-only after createWeight, copied deeper, or documented as holding shareable values is a maintainer decision.
• A caller that calls getValues without rewrite behaves as today, so stateful sources still fail there. The DoubleValuesSource javadoc requires rewrite first (DoubleValuesSource.java:34-36); consumers that skip it are separate bugs with fixes in those consumers.
• rewrite now does the per-query work (ttf and sttf loop over all segments once). Its javadoc says "Implementations should assume that this will only be called once" (DoubleValuesSource.java:94), and fromQuery already builds its Weight there (:776-780).
• That work, and any exception a custom createWeight throws, now happens when FunctionScoreQuery.createWeight rewrites the source (FunctionScoreQuery.java:121), before it is known whether the inner query matches anything; today the bridge is reached only for segments where the inner query has a scorer (:236-239).
• The rewritten wrapper's context holds the searcher; the rewritten wrapper held it before too (LUCENE-8726).
• By code reading only: QueryValueSource would find its Weight in the context instead of rebuilding it once per segment. No benchmark was run.

What the checks validated

• Gradle, stock 645d4fe14a20, seed DEADBEEF: the repro fails with assertions on and off.
• Gradle with B: :lucene:queries:test with the repro and the study's three test classes added (the finder's 5 tests, the reproducer's 7, the devil's advocate's 6): 531 tests, 0 failures, 5 skipped, of which 512 are upstream tests. ecjLintMain passes; checkGoogleJavaFormat flags only the added test files, not ValueSource.java, so that gradle invocation (run with --continue) ends BUILD FAILED while its test task is SUCCESS. The whole module ran with assertions on only.
• With -Ptests.asserts=false: the repro, the three study classes and TestValueSources, TestFunctionScoreQuery, TestFunctionScoreExplanations, TestFunctionQuerySort and TestKnnVectorSimilarityFunctions: 95 tests (76 upstream), 0 failures, 1 skipped.
• The devil's advocate's call-counting probe: with B the bridge makes 1 createWeight call for 2 segments, as FunctionQuery does (stock: 0; option A: 2).

In each of the three methods, take the rewrite-time searcher (or, if the wrapper was never rewritten, a new top-level IndexSearcher), build the context from it and call in.createWeight(context, s) before in.getValues.

IndexSearcher s = searcher != null ? searcher : new IndexSearcher(ReaderUtil.getTopLevelContext(ctx));
Map<Object, Object> context = newContext(s);
in.createWeight(context, s);

Trade-offs

• Simplest, and also covers callers that skip rewrite; that fallback builds a new IndexSearcher per segment and deserves its own review.
• Repeats the per-query work per segment: 2 createWeight calls for 2 segments (executed), and ttf and sttf loop over all segments inside createWeight, so O(segments^2) term lookups per query (code reading).

What the checks validated

• With the patched class shadowed: the repro passes with assertions on and off; the finder's 5, the reproducer's 7 and the devil's advocate's 6 tests pass; 76 upstream function tests (TestValueSources, TestFunctionScoreQuery, TestFunctionScoreExplanations, TestKnnVectorSimilarityFunctions, TestFunctionQuerySort) pass stock and with the fix. Not run through gradle.

Compute the statistic in getValues when the context has no entry, as ScaleFloatFunction and QueryValueSource do.

Trade-offs

Fixes the in-tree trigger only: a custom source that follows the createWeight javadoc stays broken, and the statistic is recomputed per segment.

Say in the bridge javadoc that sources needing createWeight are not supported.

Trade-offs

Contradicts the getValues javadoc and helps no released user; a maintainer who prefers deprecating ValueSource (#16820) might still choose it.

Method: semi-formal (bounded-exhaustive translation validation of a documented contract on the stock jar, then an executed public-API test).

• A hunt unit sweeping public APIs that no earlier hunt had named (core and queries) took the ValueSource#createWeight javadoc as a contract: every consumer must call createWeight on a context before calling getValues with it, and composites must propagate it.
• An AI finder wrote a checker that, by reflection, enumerates the 46 concrete ValueSource classes of lucene-queries, builds the 22 composites (including anonymous subclasses of the public abstract composite bases) over strict probe children, and drives each through 8 consumer entry points: 176 checks. A probe child fails when getValues receives a context its createWeight never saw.
• Under FunctionQuery, FunctionRangeQuery and ValueSource#getSortField (4 entry points) only the children of RangeMapFloatFunction's target and default sources (the sibling bug LFV-886) and of the three vector similarity functions (which do not override createWeight) saw getValues without createWeight: 8 violations per entry point. Under asDoubleValuesSource() and asLongValuesSource() (4 entry points) every composite's children did: 40 per entry point. With both fixes shadowed (this one and LFV-886's), every entry point, the bridges included, is left with the same 6 violations, the vector similarity functions' children, which this page does not analyse.
• The AI tool then reduced the signal by hand to a JUnit with public API only (ttf, sttf, a sort and a contract-following custom source), and a later step reduced that to the one-method repro. An earlier hunt unit had noted the ttf NullPointerException as an unreported lead without filing it, because ttf is @lucene.internal.