ExpressionValueSource equals/hashCode ignore the compiled functions, so the same expression text compiled with different custom functions shares query-cache entries

JavascriptCompiler.compile(String, Map<String, MethodHandle>) compiles an expression with custom functions, and its javadoc documents alternate function maps and MethodHandle#bindTo receivers. So the same text f(x) can mean abs(x) in one compilation and -x in another. ExpressionValueSource.equals and hashCode compare only the source text, the bound variables and needsScores; the compiled Expression keeps no record of its functions. The two value sources compare equal with equal hash codes, and so do the FunctionMatchQuery (or facet DoubleRange drill-down) queries built on them.

A query cache finds entries by query equality, so it serves one query's per-segment match set for the other: on 5 documents with x = -3, -2, -1, 1, 2, -x > 0 counts 5 instead of 3 after abs(x) > 0 was cached. On 7.3.0 to 10.6.0 the default IndexSearcher cache can do it (on segments of at least 10,000 documents that are large relative to the index); on main the application must install a cache.

LUCENE-8099 (2017, first released in 7.3.0) moved equality from the compiled object to the source text, which keeps recompiled identical text cacheable but is right only while one text always gets the same functions. Affected: 7.3.0 to 10.6.0 and main. Applications that compile with the default functions or one reused map are not exposed. Fix: record the functions an expression calls and compare them by identity on top of the text (validated).

1. Custom functions are documented. "You can compile with an alternate set of functions via compile(String, Map)", and "you can use MethodHandle#bindTo(Object) to bind it to a receiver" (JavascriptCompiler.java:84-102, :173). compile(String) uses the static DEFAULT_FUNCTIONS map (:154-156, :866).
2. The functions live only in the compiled class. Each called function gets an index (:471-479), and the hidden class receives the called MethodHandles as class data (defineHiddenClassWithClassData, :274-278). The Expression object keeps only sourceText and variables (Expression.java:71, :74, :82-85) and does not override equals or hashCode.
3. Equality uses only the text. hashCode hashes expression.sourceText (ExpressionValueSource.java:140), equals compares expression.sourceText.equals(other.expression.sourceText) (:162), plus needsScores and the bound variable sources (:136-172). toString is expr(<text>) (:132-134), so both queries print as FunctionMatchQuery(expr(f(x))).
4. Consumers inherit it. FunctionMatchQuery is equal when its source and its predicate are equal (FunctionMatchQuery.java:118-129), so the two queries collide when they share one predicate instance, or use distinct predicate objects whose equals returns true (lambdas have identity equality; the repro shares one positive variable). The facet DoubleRange.getQuery drill-down compares range, fast-match query and source (DoubleRange.java:118-137, :335-336) and needs no shared instance.
5. The cache trusts equality. Query.equals is "required so that QueryCache works properly", and equal queries should have identical document-filtering properties (Query.java:96-107). LRUQueryCache keys entries per segment, so the wrong answer is always the other query's match set on the same segment.

Recording the called functions and comparing them by identity, and nothing else, makes the repro pass; in the history step, reverting only the two equality lines on 7.3.0 makes it pass too. So the equality is the cause.

Controls with different expression text pass.

• Silent. The second query returns the first query's match set, with no exception, and both print as FunctionMatchQuery(expr(f(x))).
• Which applications. The optional expressions module; the same text compiled with different Map<String, MethodHandle> maps or with one method bound to different receivers (bindTo), for example per tenant or per request; equal variable bindings; and an equality-keyed consumer: FunctionMatchQuery with a shared predicate instance (or predicates that compare equal), or a facet DoubleRange drill-down. The class javadoc describes binding a function to a receiver with bindTo (JavascriptCompiler.java:99-102), and uschindler gave it as a "Small tip" on java-user@ (2025-05-07, in a thread that does not discuss caching). The compile(String, Map) javadoc says instead "Functions must be {@code public static}" (:162); the compiler accepts bound handles anyway (:904-925). The headline repro uses two public static methods and does not need bindTo.
• Not exposed. Compiling with the default functions or one reused map: the same text then always gets the same functions. In the code paths checked (not plugins or older versions), Elasticsearch, OpenSearch and Solr compile expressions with the default functions. A handle bound to a receiver whose state changes is not covered by text equality, nor by fix A. Scores through LRUQueryCache are not affected: it caches only uses that need no scores (IndexSearcher.java:976-981), so a top-level scoring FunctionMatchQuery is never cached, while the same query in a FILTER clause or a count can be.
• Cache reach by version. 7.3.0 to 10.6.0: every IndexSearcher starts with a JVM-wide LRUQueryCache(1000, min(32 MB, heap/20)) and UsageTrackingQueryCachingPolicy; it caches segments with at least 10,000 documents that are also large relative to the index (at least 3% of it up to 9.5, more than half the average segment size from 9.6.0), once a plain filter has been seen 5 times, if the value source is cacheable and the use needs no scores. Because the hash codes are equal, the two queries' uses add up. Executed with stock 10.5.2 defaults: 16000 instead of 8000. Main: "Caching is disabled by default." (GITHUB#14187, Lucene 11.0.0, unreleased), so the application must install a cache.
• Likelihood: rare. A rarely used API shape, and the documented searches found no public report in the eight years since 7.3.0.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<3> but was:<5>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:647)
       org.junit.Assert.assertEquals(Assert.java:633)
       org.apache.lucene.expressions.TestExpressionFunctionsQueryCacheKey.testSameTextDifferentFunctionThroughQueryCache(TestExpressionFunctionsQueryCacheKey.java:87)

• The query contract. Query.equals exists "so that QueryCache works properly"; "Typically a query will be equal to another only if it's an instance of the same class and its document-filtering properties are identical to those of the other instance" (Query.java:96-107).
• Value-source equality is meant to track values. In LUCENE-7736 (#8787) jpountz wrote "I think it's important for equals to behave correctly and never return true for values sources that could return different values". DoubleValuesSource.equals/hashCode are abstract (DoubleValuesSource.java:126-129, LUCENE-7723 / #8774).
• The introducing change chose expressions for their trustworthy equality. In LUCENE-8099 (#9147) romseygeek wrote that a closure-based source "messed up equality tests (essentially, you can't check Java closures for equality). I think expressions are a good solution? They allow for hashcode/equals tests". A function bound with bindTo is such a closure.
• Precedent. LUCENE-5220 (#6284, rmuir, 2013): ExpressionSortField "will bogusly report equals(true) if the expression is the same: but bindings could be different!". That was variable bindings in a class deleted in 2017; this is function bindings.
• History. Up to 7.2.1, equals compared the compiled Expression objects, which have identity equality. LUCENE-8099 (b01e6023e1cd) switched both lines to sourceText; its commit message, CHANGES entry and issue do not mention the change or function maps. Reading it as "keep recompiled identical text cacheable" is an inference.
• No documentation, no test. No javadoc says equality ignores the functions; TestExpressionValueSource.testDoubleValuesSourceEquals compiles once and reuses one Expression (TestExpressionValueSource.java:100-132).

The strongest case for "intended" (the cross-model review and the intended-behaviour check): text equality was chosen on purpose, so that recompiling the same text stays equal and cacheable; Expression and JavascriptCompiler are @lucene.experimental (Expression.java:66, JavascriptCompiler.java:111); DoubleValuesSource.equals has no detailed contract; and main has no default query cache.

Why it fails: that purpose covers recompiling text with the same functions, and the recommended fix keeps it; different function maps and bindTo receivers are documented features nobody discussed; @lucene.experimental warns about API changes, not wrong results; and the one stated contract, query equality for the query cache, is broken, with default searcher settings on 7.3.0 to 10.6.0. Verdict: not intended, a defect (all verifiers, the refute review and the adjudication; the adjudication with high confidence).

Two fixes were built and validated through gradle on 645d4fe14a20 (scratch worktree, OpenJDK 25.0.3, seed DEADBEEF, 2026-10-31), both with the test-only build.gradle line the regression test needs. Workarounds until a fix ships: give each function variant its own name, pass per-tenant values as variable bindings (which equality compares) instead of bindTo receivers, or do not cache such queries.

Expression gets a package-private final List<MethodHandle> functions and a new protected constructor Expression(String, String[], List<MethodHandle>) that stores List.copyOf(functions); the existing 2-argument constructor passes List.of(), so hand-written subclasses behave as today. JavascriptCompiler's generated constructor takes the list of called MethodHandles it already hands to defineHiddenClassWithClassData and passes it on. ExpressionValueSource compares and hashes the list. MethodHandle does not override equals/hashCode, so the handles are compared by identity; DEFAULT_FUNCTIONS and a reused map hold the same instances across compilations.

// Expression.java
final List<MethodHandle> functions;
protected Expression(String sourceText, String[] variables, List<MethodHandle> functions) {
  ...  this.functions = List.copyOf(functions);
}
// JavascriptCompiler.java: the generated constructor passes the called MethodHandles on
// ExpressionValueSource.java
result = prime * result + ((expression == null) ? 0 : expression.functions.hashCode());
...
} else if (!expression.functions.equals(other.expression.functions)) {
  // MethodHandle has identity equality: the same text compiled with other functions differs
  return false;
}

Trade-offs

One API addition: a protected constructor on the public @lucene.experimental Expression class. Two conservative cases become unequal: the same receiver bound twice with bindTo, and the same static method looked up afresh with findStatic for each compilation. They cost cache hits, never correctness; they were observed on JDK 25 (distinct handles from two lookups or two bindTo calls are not an API guarantee). Applications that build handles per request lose cache reuse; how common that is is unknown. Default functions, one reused map, and one Expression with two value sources stay equal (executed scratch check; on stock all 7 cases compare equal).

What the checks validated

• The repro fails on stock and passes with the fix, assertions on and off.
• The whole :lucene:expressions:test: 14 suites, 171 tests (170 upstream + the repro), 0 failures.
• :lucene:queries:test --tests 'org.apache.lucene.queries.function.*' (includes TestFunctionScoreQuery, which uses expressions): 111 tests, 0 failures.
• :lucene:expressions:check -x test (google-java-format, ecj lint, forbidden APIs) passes after tidy of the test file; renderJavadoc, and the compile of benchmark-jmh (which subclasses Expression) and demo, pass.

In ExpressionValueSource.equals, replace the sourceText comparison with expression != other.expression (1 line; hashCode can stay text-based). The finder's sketch is a variant that keeps text equality for hand-written subclasses.

Trade-offs

Every recompilation is unequal, even of the same text with the default functions, which undoes what LUCENE-8099 apparently wanted: fewer cache hits for applications that recompile per request. No API change.

What the checks validated

• The repro passes, assertions on and off; the whole :lucene:expressions:test passes (172 tests, including the repro and the scratch check). In the study, the finder's variant passed all 170 expressions tests.

State in the JavascriptCompiler and Expression javadoc that value-source equality uses only the text and the variables.

Trade-offs

No behaviour change: cached results stay wrong for the exposed applications.

What the checks validated

Not built.

Method: non-formal (a hunt unit, expressions compiler).

• During the unit's secondary contract review of ExpressionValueSource.equals/hashCode, the finder, an AI tool, noticed that equality uses the source text while the values depend on the function map the text was compiled with.
• It wrote a deterministic public-API JUnit on the stock main jars (5 documents, an always-caching LRUQueryCache), which fails with assertions on and off, and a fix sketch that makes it pass.
• Independent verifiers then wrote their own tests from the claim alone (a 20,000-document cache set-up, the bindTo form, facet drill-downs). The one-test repro was cut from the finder's test on 2026-10-29.