RangeMapFloatFunction (map()) never calls createWeight on its target and default, so ttf, sttf or any ValueSource that keeps per-query state there throws NullPointerException

RangeMapFloatFunction is the map() function query: a document whose source value lies in [min, max] gets the target's value, any other document the default's. Since 4.7.0 the target and default may be any ValueSource. Some ValueSources, in-tree TotalTermFreqValueSource (ttf) and SumTotalTermFreqValueSource (sttf), compute their value once per query in createWeight and keep it in the per-query context, where getValues reads it back, as the ValueSource javadoc describes. map() reads all three children on that context but passes createWeight only to the source, so ttf or sttf as the target or default returns null and the first document that takes that branch throws NullPointerException; a custom ValueSource that keeps per-query state finds its entry missing. One document is enough: FunctionQuery(map(1, 0, 2, sttf(text))) should score 2 and throws, with assertions on and off and the default configuration; every release from 4.7.0 to 10.6.0 and main is affected (14 versions run, the releases in between by reading). The fix is four lines that forward createWeight to the target and the default, as every other non-vector composite does.

1. FunctionQuery.FunctionWeight creates the context and calls createWeight on the root (FunctionQuery.java:70-71); each segment's scorer then calls func.getValues(weight.context, context) (:116). A ValueSource sort does the same in ValueSourceSortField.rewrite (ValueSource.java:379-381), and so does FunctionRangeQuery (FunctionRangeQuery.java:148-149).
2. RangeMapFloatFunction.createWeight calls only source.createWeight(context, searcher) (valuesource/RangeMapFloatFunction.java:105-108).
3. RangeMapFloatFunction.getValues reads all three children on that context: source (:75), target (:76) and, when non-null, defaultVal (:77-78). floatVal returns targets.floatVal(doc) for an in-range value (:83-84), else defaults.floatVal(doc) (:85).
4. SumTotalTermFreqValueSource.createWeight sums the field's token count over all segments and stores it with context.put(this, ...) (valuesource/SumTotalTermFreqValueSource.java:57-74); its getValues is return (FunctionValues) context.get(this); (:51-54). TotalTermFreqValueSource works the same way (:59-62, :65-81). Without the call the entry is missing, targets is null, and line 84 throws.

• ValueSource.newContext already puts the searcher in the context (ValueSource.java:74-77), so docfreq, maxdoc and norm, which only read it, work as targets. QueryValueSource (query(...)) and ScaleFloatFunction (scale(...)) build their state lazily inside getValues when it is missing, so they do not crash.
• In-tree, only ttf and sttf fail, and any composite that wraps one of them, because map() never calls the wrapper's createWeight either (run: scale(sttf) and sum(sttf, const(0)) as the target throw, sum(const(3), const(4)) scores 7); plus any custom ValueSource that stores per-query data in createWeight as the javadoc describes.
• Masking: the context is an IdentityHashMap (ValueSource.java:75) and ttf/sttf key their entry by this. If the very same ttf/sttf object is both map's source and its target, the source's createWeight fills that object's entry and the bug is hidden; equal but distinct instances do not hide it.

• One document "x x", map(1, 0, 2, sttf(text)): expected score 2; NullPointerException "this.val$targets" is null at line 84 (main, -ea and -da; 10.6.0 release jars, -ea and -da).
• Two documents, ttf(text, test) as the target: the same NPE; sttf(text) as the default of an out-of-range value: NPE "this.val$defaults" is null at line 85.
• A custom ValueSource that stores the document count in createWeight, as target or as default: its getValues finds no entry.
• A ValueSource sort by the same map(...) after Sort.rewrite (new Sort(map.getSortField(true)).rewrite(searcher); reverse false and true both run): the same NPE, from ValueSourceComparator.copy.
• FunctionRangeQuery over the same map(...), and IndexSearcher.explain of its FunctionQuery: the same NPE at line 84; both work with the fix.
• Controls on the same index pass: ttf, sttf and the custom source alone; IfFunction with the same stateful children; a stateful source as map's first argument.

• map(x, min, max, query(...)) does not crash: QueryValueSource builds its Weight inside getValues and writes it into the shared context, which adds one writer to a separate, known context-map race of concurrent searches (LFV-567).
• The three vector-similarity composites (through VectorSimilarityFunction, which has no createWeight override) skip both children too, and the propagation harness flags them, but no in-tree vector ValueSource keeps per-query state. Every "only" or "every other" on this page means the non-vector composites.

• Who is hit. Callers of RangeMapFloatFunction with a ValueSource target or default that keeps per-query state set in createWeight: in-tree TotalTermFreqValueSource (ttf) and SumTotalTermFreqValueSource (sttf), both public classes marked @lucene.internal, or a custom ValueSource that follows the createWeight javadoc. Consumers that call createWeight on the root fail: FunctionQuery scoring and a ValueSource sort after Sort.rewrite (both run). The hunt's propagation harness also flags FunctionQuery.explain and FunctionRangeQuery for the same two children, and the fact check ran both on the repro's index: the same NPE at line 84.
• Who is not. map() with constants, fields, docfreq, maxdoc, norm or query(...) as target or default; arithmetic or scale(...) whose whole child tree needs no createWeight (sum(sttf, ...) and scale(sttf, ...) fail); and stateful sources as the mapped (first) argument.
• Default vs opt-in. Default IndexWriterConfig and a plain IndexSearcher, single thread; the main() programs and the diagram probe use Lucene's stock default codec, the JUnit runs the codec the test framework picks from the seed, and all fail the same way. The API is opt-in, and FunctionQuery is @lucene.experimental.
• Scale. One document is enough. With ttf or sttf as the target or the default, every search throws as soon as one document takes that branch (in-range values read the target, the others the default); if no document takes it, nothing fails. For ttf and sttf the failure is loud and limited to that query, with no index damage. A custom ValueSource may throw, build its state lazily, or fall back to some value when its entry is missing; a fallback would give a wrong score silently (plausible, not tested).
• How likely. Rare: most map() calls use constants or fields, and the searches found no report since 4.7.0 (2014).

The test indexes one document with TextField("text", "x x") in a ByteBuffersDirectory and runs new IndexSearcher(reader).search(q, 1) with q = FunctionQuery(RangeMapFloatFunction(ConstValueSource(1), 0, 2, SumTotalTermFreqValueSource("text"), null)). Single thread, deterministic for a given seed; the test draws no random values. The test framework still picks the codec, similarity, locale and time zone from the seed (the DEADBEEF run used codec=Asserting(Lucene104)); the failure does not depend on them: it fails with every seed run, and the main() programs on Lucene's stock default codec fail the same way. The NullPointerException is caught once and rethrown as an AssertionError so that the message states expected against actual; the NPE is kept as the cause. The test needs the org.apache.lucene.tests framework (9.1.0 and later); older releases were checked with an equivalent main() program.

TestRangeMapTargetCreateWeight > testStatefulTarget FAILED
    java.lang.AssertionError: expected score 2, got java.lang.NullPointerException: Cannot invoke "org.apache.lucene.queries.function.FunctionValues.floatVal(int)" because "this.val$targets" is null
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:DC1CD76A43A01F29]:0)
        at org.apache.lucene.queries.function.TestRangeMapTargetCreateWeight.testStatefulTarget(TestRangeMapTargetCreateWeight.java:55)

        Caused by:
        java.lang.NullPointerException: Cannot invoke "org.apache.lucene.queries.function.FunctionValues.floatVal(int)" because "this.val$targets" is null
            at org.apache.lucene.queries.function.valuesource.RangeMapFloatFunction$1.floatVal(RangeMapFloatFunction.java:84)
            at org.apache.lucene.queries.function.FunctionQuery$AllScorer.score(FunctionQuery.java:131)
            at org.apache.lucene.core@11.0.0-SNAPSHOT/org.apache.lucene.search.TopScoreDocCollector$1.collect(TopScoreDocCollector.java:95)
            ...
            at org.apache.lucene.core@11.0.0-SNAPSHOT/org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:624)
            at org.apache.lucene.queries.function.TestRangeMapTargetCreateWeight.testStatefulTarget(TestRangeMapTargetCreateWeight.java:53)
1 test completed, 1 failed

# the same with -Ptests.asserts=false and on the 10.6.0 release jars; with the fix the test passes

• The contract. ValueSource.java:43-44: getValues "Gets the values for this reader and the context that was previously passed to createWeight()". :65-68: "Implementations should propagate createWeight to sub-ValueSources which can optionally store weight info in the context. The context object will be passed to getValues() where this info can be retrieved."
• The siblings. IfFunction.createWeight calls all three children (valuesource/IfFunction.java:155-159), and so do DualFloatFunction, MultiFloatFunction, MultiFunction, MultiBoolFunction, ComparisonBoolFunction and the single-child composites. A harness over all 46 concrete ValueSource classes, 22 composites and 8 consumers (176 checks) found RangeMapFloatFunction the only non-vector composite that skips a child. With this fix and the separate bridge fix (LFV-893) both shadowed it reports no non-vector violation; the 48 left are all the three vector-similarity composites, which skip both children too and are not claimed.
• The promise. lucene/CHANGES.txt:12745: "SOLR-1871: The RangeMapFloatFunction accepts an arbitrary ValueSource as target and default values."
• The history. createWeight was written by SOLR-1432 (2009) when target and default were float constants, so forwarding to the source was complete. SOLR-1871 (4109c6db336, 2013-12-02) made them ValueSources and updated getValues, equals, hashCode and description, but not createWeight. Its JIRA thread never mentions createWeight, and no commit ever added the forwarding.
• The tests. TestValueSources.testRangeMap (:564) uses only constant, field and sum operands, which need no createWeight; testSumTotalTermFreq (:613-617) and testTotalTermFreq (:686-690) score sttf and ttf through FunctionQuery, as legal inputs.

The strongest case for "works as designed": the javadoc says "should", not "must", and the base createWeight is a no-op, so a ValueSource that depends on it is fragile; the in-tree triggers are @lucene.internal; QueryValueSource and ScaleFloatFunction cope lazily, which looks like a convention; FunctionQuery is experimental and an open issue (#16820) proposes deprecating ValueSource for Lucene 11; and the class is not final, so a user can override createWeight.

Why it fails: "should" does leave some design discretion (and the vector-similarity composites show that propagation is not enforced everywhere), but the getValues javadoc itself says it gets the context "previously passed to createWeight()", which map() breaks for two of its three children; a contract-following custom ValueSource fails through public constructors alone; every other non-vector composite forwards; nothing documents that children must tolerate a skipped createWeight; the SOLR-1871 diff shows an oversight; and ValueSource is not deprecated, while 9.x and 10.x are affected. A subclass that overrides createWeight is a workaround, not a contract. Verdict: not intended, a defect.

Option A was applied in a scratch worktree of 645d4fe14a20 and run with gradle (the worktree was removed after the runs); the hunt's earlier checks validated the same diff. Options B and C were not built as fixes.

+4 lines in RangeMapFloatFunction.createWeight (valuesource/RangeMapFloatFunction.java:105-108): also call target.createWeight and, when defaultVal is not null, defaultVal.createWeight.

public void createWeight(Map<Object, Object> context, IndexSearcher searcher) throws IOException {
  source.createWeight(context, searcher);
  target.createWeight(context, searcher);
  if (defaultVal != null) {
    defaultVal.createWeight(context, searcher);
  }
}

Trade-offs

• Restores the getValues contract for both children and matches every other non-vector composite. Stateless children are unaffected (their createWeight is the base no-op). defaultVal may be null, hence the guard.
• Cost: one createWeight call per child occurrence per query, not per document, as in the sibling composites. It runs even when no document takes that branch, and a child object that appears in two positions gets it twice (a repeated ttf/sttf sums its statistic again; a repeated query(...) overwrites its Weight entry).
• Side effect worth naming: with query(...) as target or default, QueryValueSource now builds its Weight in createWeight instead of lazily inside getValues, which removes that lazy write into the shared context (the separate context-map race, LFV-567, remains).

What the checks validated

• Gradle, stock 645d4fe14a20 (seed DEADBEEF): the repro fails with assertions on and off.
• Gradle with the fix, assertions on: the repro and all 14 upstream test classes of org.apache.lucene.queries.function (including TestValueSources, TestFunctionQuerySort, TestFunctionRangeQuery and TestFunctionQueryExplanations): 15 classes, 112 tests (111 upstream), 0 failures, 0 skipped.
• Gradle with the fix, -Ptests.asserts=false: the repro and TestValueSources, 42 tests, 0 failures; :lucene:queries:checkGoogleJavaFormat passes.
• The standalone runner with the fixed class shadowed into the stock main jars: the repro passes with -ea and -da. The hunt's earlier checks validated the same diff: the finder's 4 tests and an independent 8-test suite pass, and with this fix and the separate bridge fix (LFV-893) both shadowed, the 176-check harness reports no violation for RangeMapFloatFunction or any other non-vector composite (48 vector-similarity violations remain, not claimed).

The class is not final and its fields are protected: a subclass can override createWeight the same way. The version probes ran exactly this as a control group, and it passes on every version run.

Trade-offs

Helps users on old releases; leaves the bug in place for everyone else.

Like ScaleFloatFunction, compute the statistic inside getValues when createWeight was skipped and cache it in the shared context (ScaleFloatFunction computes its ScaleInfo once, on the first getValues call, and stores it there).

Trade-offs

Fixes only the two in-tree triggers; custom ValueSources that follow the javadoc still fail.

Method: semi-formal (bounded-exhaustive check of one API contract on the stock jars).

• While writing the reference for the ValueSource#createWeight propagation contract, the finder noticed that RangeMapFloatFunction.createWeight reaches only source.
• A harness then used reflection over all 46 concrete public ValueSource classes of lucene-queries, built 22 composites (including anonymous subclasses of the 6 abstract composite bases) with strict probe children that flag getValues on a context their createWeight never saw, and ran each under 8 consumer entry points: 176 checks. Apart from the three vector-similarity composites, RangeMapFloatFunction was the only composite that skipped children under the consumers that call createWeight. With this fix and the separate bridge fix (LFV-893) both shadowed, no non-vector composite is flagged; the vector-similarity violations remain and are not claimed.
• The case was reduced to a public-API JUnit with ttf, sttf and a custom stateful source, and later to the one-document test.