scale() writes to the context map that all slices of an executor search share: NullPointerException or silently different scores

FunctionQuery, FunctionRangeQuery and ValueSource sort fields create one context map per search (ValueSource.newContext, an IdentityHashMap documented as "non-threadsafe") and hand it to every slice of an executor search. ScaleFloatFunction stores nothing in createWeight; it computes the whole-index min/max on first use in getValues and puts it into that shared map from a slice thread. When that put makes the map grow (the 22nd, 43rd or a later growth point), the other slices can briefly find entries missing: a query() that misses both its weight and "searcher" rebuilds it with the default Similarity, so a searcher whose Similarity scores differently silently gets different scores and top hits, and sources that need "searcher" (maxdoc(), idf(), ...) throw NullPointerException. In the repro (1,600 documents, 7 slices, sum(scale(num), 20 x query()), BM25Similarity(1.8, 0.3)), 20 of 20 runs return a different top 10 within the first 70 searches. Every release since 4.0.0-ALPHA is affected; the fix computes the scale in createWeight.

// ValueSource.java
/** Returns a new non-threadsafe context map. */                                 // :73
public static Map<Object, Object> newContext(IndexSearcher searcher) {
  Map<Object, Object> context = new IdentityHashMap<>();                         // :75, default capacity 32
  context.put("searcher", searcher);                                             // :76
// FunctionQuery.FunctionWeight: one context per Weight, shared by every slice
this.context = ValueSource.newContext(searcher);                                 // :70
func.createWeight(context, searcher);                                            // :71, query() puts its weight (QueryValueSource.java:85)
vals = func.getValues(weight.context, context);                                  // :116, AllScorer, on each slice thread

// valuesource/ScaleFloatFunction.java
ScaleInfo scaleInfo = (ScaleInfo) context.get(ScaleFloatFunction.this);          // :106
if (scaleInfo == null) scaleInfo = createScaleInfo(context, readerContext);      // :107-109, during the search
context.put(ScaleFloatFunction.this, scaleInfo);                                 // :98, a write to the shared map
public void createWeight(...) { source.createWeight(context, searcher); }        // :148-151, nothing stored

// valuesource/QueryValueSource.java, QueryDocValues constructor
Weight w = fcontext == null ? null : (Weight) fcontext.get(vs);                  // :114, may read null
weightSearcher = (IndexSearcher) fcontext.get("searcher");                       // :120, may read null
if (weightSearcher == null) weightSearcher = new IndexSearcher(ReaderUtil.getTopLevelContext(readerContext)); // :121-122, default Similarity

• FunctionRangeQuery shares its map the same way (FunctionRangeQuery.java:148-149, :154, :170), and so does a sort by ValueSource.getSortField (ValueSource.java:379-382, :434).
• After createWeight the map in the repro holds 21 entries: "searcher" and the 20 query() weights. The JDK's IdentityHashMap grows its table when an entry would make 3 * size exceed the table length, so at the 22nd entry for the default capacity 32 (table length 64), then at the 43rd, the 86th and so on, as the table doubles (JDK 25 IdentityHashMap.java:160, :456).
• resize copies the entries one by one into a new array, nulling each old slot as it goes (:489-501), and only then publishes the new table (:502). A get on the old table that meets a nulled slot returns null (:337-349). By reading the code, two slices that resize at the same time could also drop entries for the rest of the search (not observed directly).
• A missing "searcher" makes MaxDocValueSource throw (MaxDocValueSource.java:49-50; the same read in DocFreqValueSource.java:60, IDFValueSource.java:51, NormValueSource.java:66, TFValueSource.java:54). A query() that misses both its weight and "searcher" rebuilds its weight with the default Similarity and puts it back into the shared map (QueryValueSource.java:125, then :85), where later leaves of the same search find it; a weight lost again on the way throws NullPointerException: ... because "this.weight" is null (QueryValueSource.java:150).
• ScaleFloatFunction is the only built-in ValueSource that writes the context outside createWeight (the query() fallback above calls createWeight during a search only after an entry went missing). asDoubleValuesSource is not affected: it makes a new context per leaf (ValueSource.java:200).

1,600 documents in 32 segments (default slicing: 7 slices), an 8-thread executor, sum(scale(num, 0, 1), query(body:w0..w19)), BM25Similarity(1.8, 0.3), top 10. Sequential: 612=4.680546 1271=4.6782455 1303=4.6594014 .... The executor search differed within the first 70 of 1,000 searches in 20 of 20 runs (10 with assertions on, 10 off); in 9 of them the wrong top 10 equals, value for value, the sequential result with the default BM25Similarity(): 1271=5.1422386 612=5.1421137 1303=5.1220636 .... Other runs mix the two Similarities term by term.

• Who is exposed. Applications that use IndexSearcher(IndexReader, Executor) and run a FunctionQuery, FunctionRangeQuery or ValueSource sort whose function contains scale() and enough context-storing sources (query(), totaltermfreq(), sumtotaltermfreq(); "searcher" counts once) that scale()'s entry is the 22nd or 43rd (or a later growth point) of the map: about 20 such sources with one scale(), or 41.
• Nearby sizes. With several scale() calls the window widens. With 19 query() sources (the put is then the 21st entry, below the growth point) one of five runs also failed, at search 597 of 1,000; that failure was not instrumented and its cause is open.
• What they see. With a Similarity that scores differently from the default BM25Similarity(), silently different scores and top hits (no exception; this doc's test, 20 of 20 runs). With maxdoc(), idf(), docfreq(), norm() or tf() in the function, a failed search (earlier tests): NullPointerException: Cannot invoke "org.apache.lucene.search.IndexSearcher.getIndexReader()" because "searcher" is null. With the default Similarity, no run showed different scores (5 runs of this function), but one threw.
• Contained. Only the triggering search is affected; other searches on the same searcher and executor were unaffected (0 of about 17,000 in a probe), nothing is cached and the index is not touched.
• Not affected. Searchers without an executor; in the controls, a 10-source function (5 of 5 runs pass all 1,000 searches); asDoubleValuesSource.
• How likely. Narrow: it takes an executor and a function sized to a growth point, and the searches found no report of it since 4.0.0-ALPHA (2012). But a deployment whose ranking function sits at a growth point fails in most runs.

One test method, public API only; it sits next to the other function-query tests. The index and the function are fixed (new Random(7)); the test framework still randomizes the codec. It stops at the first executor search whose top 10 differs from the sequential one.

Regression test: this same test. It passes with the fix (gradle, assertions on and off; 10 of 10 standalone runs) and passes :lucene:queries:check -x test. It covers FunctionQuery and the silent-score form only: the FunctionRangeQuery, sort-field and maxdoc() NullPointerException forms rest on the earlier tests (see Independent checks).

Hit rate: on a 32-core aarch64 machine, 20 of 20 standalone runs fail (seeds 5CA1E001..5CA1E010, assertions on and off), the first wrong search between search 1 and search 68; gradle fails too, although in this run Lucene's gradle setup made the test JVM report 1 CPU (it adds -XX:ActiveProcessorCount=1 outside CI when default vectorization is off); the 8 executor threads still run in parallel. On a machine with few cores the race may need more searches to show.

Controls (5 runs each, assertions on; the two source-count controls change one line, the default-Similarity control both setSimilarity lines): 10 query() sources: 5 of 5 pass. 19 sources: 4 of 5 pass, one failed at search 597 (not instrumented; the cause is open). Default BM25Similarity(): no different scores in these 5 runs, but 1 of 5 threw NullPointerException ... "this.weight" is null.

TestScaleFunctionWithExecutor > testExecutorSearchMatchesSequentialSearch FAILED
    org.junit.ComparisonFailure: search 22 expected:<[612=4.680546 1271=4.6782455 1303=4.6594014 24=4.650218 1480=4.630164 309=4.6299486 764=4.6039133 348=4.5888166 1060=4.5771465 1527=4.524094] > but was:<[1271=5.1422386 612=5.1421137 1303=5.1220636 24=5.108552 1480=5.0892344 309=5.0861616 764=5.069854 348=5.0580726 1060=5.035736 1527=4.9915533] >
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:AD787DCA0434352F]:0)
        at app/junit@4.13.2/org.junit.Assert.assertEquals(Assert.java:117)
        at app//org.apache.lucene.queries.function.TestScaleFunctionWithExecutor.testExecutorSearchMatchesSequentialSearch(TestScaleFunctionWithExecutor.java:87)
:lucene:queries:test (FAILURE): 1 test, 1  failure

# assertions on (same seed): search 11 returned a mixed result, 1271=5.1422386 612=4.9920917 1060=4.9601364 ...
# supporting probe, sequential only, same index and function:
#   BM25Similarity(1.8, 0.3): 612=4.680546 1271=4.6782455 1303=4.6594014 ...
#   BM25Similarity()        : 1271=5.1422386 612=5.1421137 1303=5.1220636 ...

• Contract. IndexSearcher instances "are completely thread safe, meaning multiple threads can call any of its methods, concurrently" (IndexSearcher.java:71-75); the executor constructor "Runs searches for each segment separately, using the provided Executor" (:194). Nothing says function queries need a searcher without an executor.
• ValueSource's convention. "Implementations should propagate createWeight to sub-ValueSources which can optionally store weight info in the context. The context object will be passed to getValues() where this info can be retrieved." (ValueSource.java:66-68). ScaleFloatFunction stores nothing in createWeight (:148-151) and writes in getValues (:98, :106-109).
• Who shares the map. FunctionWeight, FunctionRangeWeight and ValueSourceSortField hand the one map to every slice; the user never sees it.
• Tests. TestValueSources searches through newSearcher(reader) (TestValueSources.java:201), which randomly adds an executor (LuceneTestCaseParent.java:2563-2583, :2641-2675); its testScale (:593) passes because its functions are small.
• Effect. The configured Similarity is silently replaced by the default one for some or all of the terms of a search; that is not a documented trade-off of concurrent search.

Strongest case for "intended": newContext is documented "Returns a new non-threadsafe context map" (ValueSource.java:73); FunctionQuery (FunctionQuery.java:38) and the executor constructor (IndexSearcher.java:199) are @lucene.experimental; and 20 query() sources is an unusual function.

Why it fails: that javadoc warns whoever shares the map, and the sharing is Lucene's own Weight code; experimental status does not make silently different scores acceptable; no limit on function size is documented, and the failures appear at or next to the map's growth points (21 or 42 entries before scale()), not at some size where things degrade. The intended-behaviour verifier and reviews by another model family (OpenAI GPT-5.6, reading-only) rejected this argument. Verdict: not intended, a defect.

Fix A was validated with gradle in a scratch worktree of 5adff78bb168; B was checked earlier with the changed class compiled over the jars in standalone runs; C was not run. A fixes scale(); B also covers custom ValueSources that write the context from getValues. They combine well.

ScaleFloatFunction.createWeight computes the min/max right after source.createWeight, on the thread that creates the Weight, and stores it in the context; getValues then only reads it. createScaleInfo takes an IndexReaderContext instead of a LeafReaderContext, so it can be called with searcher.getTopReaderContext(); ReaderUtil.getTopLevelContext walks to the same root as the stock lazy path, so the min/max is unchanged. The lazy path stays for callers that never call createWeight (asDoubleValuesSource, one context per leaf).

private ScaleInfo createScaleInfo(Map<Object, Object> context, IndexReaderContext readerContext)
    throws IOException { /* unchanged body */ }

@Override
public void createWeight(Map<Object, Object> context, IndexSearcher searcher) throws IOException {
  source.createWeight(context, searcher);
  // The slices of a concurrent search share this context, so compute the scale now: getValues
  // must only read the context, never write to it.
  if (context.get(ScaleFloatFunction.this) == null) {
    createScaleInfo(context, searcher.getTopReaderContext());
  }
}

Trade-offs

• Fixes scale() only: a custom ValueSource that writes the context in getValues stays unsafe (see B).
• The whole-index scan moves into createWeight: it runs once, on the calling thread, before the slices start (today each slice that misses the entry may scan in parallel), and it also runs when no scorer is ever pulled. TotalTermFreqValueSource.createWeight already loops over all leaves (TotalTermFreqValueSource.java:65-71), an in-tree precedent for the eager scan.
• No targeted test yet covers a nested scale() or the FunctionRangeQuery and sort-field entry points; the runs cover this test and the existing suites only.

What the checks validated

• The repro passes with assertions on and off; with the fixed class compiled over the jars in standalone runs, 10 of 10 runs pass (5 on, 5 off).
• All 15 classes under org.apache.lucene.queries.function, 112 tests including TestValueSources.testScale and the repro, pass; the whole :lucene:queries:test, 513 tests, 5 skipped, passes.
• :lucene:queries:check -x test passes (google-java-format, ecj lint, forbidden APIs) with the fix and the new test.

ValueSource.newContext returns Collections.synchronizedMap(new IdentityHashMap<>()). This covers every writer, custom sources included.

Map<Object, Object> context = Collections.synchronizedMap(new IdentityHashMap<>());

Trade-offs

• A lock on each context access (per leaf and source, not per document).
• Slices may still compute the scale twice (harmless, same value).
• A caller-supplied map, for example in GroupingSearch(ValueSource, Map), is not covered.

What the checks validated

Checked earlier with a replaced ValueSource on main ea57360912ed: the independent reproducer's four tests pass with assertions on and off, 0 of 20 + 0 of 20 runs fail on a 64-vCPU host, and the same replacement passes on 4.0.0. Not run through gradle.

Copy the context per slice or leaf after createWeight. Not recommended alone: with the lazy scale() each copy would rescan the whole index. Only useful together with A.

Trade-offs

One small map copy per leaf; needs A to avoid repeated scans.

What the checks validated

Not run.

Method: semi-formal (2026-10-26, main ea57360912ed).

• A bytecode scan (an abstract interpretation of the class files with a per-basic-block symbolic operand stack that resolves the receiver of every field write and collection mutation) covered 689 subtypes of Weight, Query, ValueSource, Similarity and other search-time types in 17 jars. It flagged ScaleFloatFunction.createScaleInfo as the only built-in ValueSource method that puts into the per-search context map outside createWeight. Code reading showed that FunctionWeight hands that map to every slice.
• A differential test of executor and sequential searches passed for small functions, so a bounded-exhaustive model of the real JDK growth rule enumerated every pair of entries before the search (up to 120) and puts during the search (up to 8): 968 cases, 108 of them grow the map during the search.
• Directed experiments and sweeps on a 64-vCPU host then failed at the predicted growth points (20 and 41 padding sources: 5 of 5 runs each; none at 18, 19, 21, 22 and 42 in the independent reproducer's sweep; one throw in five runs at 40, next to the second).