Vector similarity value sources: equals() ignores the similarity function that hashCode() includes, so EUCLIDEAN and COSINE sources compare equal

FloatVectorSimilarityFunction, ByteVectorSimilarityFunction and (on main and branch_11x) Float16VectorSimilarityFunction are lucene/queries value sources that score each document by similarityFunction.compare(v1, v2), for example a document's vector field against a constant query vector. Their shared base class valuesource.VectorSimilarityFunction holds three fields, but its equals() compares only the two vector sources, while hashCode() (Objects.hash(similarityFunction, vector1, vector2)) and description() include the function. So a EUCLIDEAN and a COSINE source over the same field and query vector, which score a document [2, 1] 0.333 and 0.9, compare equal with different hash codes, breaking the java.lang.Object contract; FunctionQuery, FunctionRangeQuery and FunctionScoreQuery pass the false equality on. Inside Lucene no search result changes in practice, because these value-source weights are never cached and query de-duplication is hash-based (only a collision of the two enum constants' identity hash codes could defeat that); only callers that compare with equals alone are misled. It has been in every release since the classes were added in 9.7.0 (#12253): an earlier commit of that PR compared the function, and a review commit dropped it. The fix is one added condition.

1. Three fields: similarityFunction, vector1, vector2 (:29-31), set by the public constructor (:33-41).
2. The function decides the values. Each subclass's func ends in return similarityFunction.compare(v1, v2); (FloatVectorSimilarityFunction.java:46, ByteVectorSimilarityFunction.java:47, Float16VectorSimilarityFunction.java:46). No subclass overrides equals or hashCode.
3. equals leaves it out (:79-85): after the this == o, null and getClass() checks it returns Objects.equals(vector1, ...vector1) && Objects.equals(vector2, ...vector2).
4. hashCode and description() include it: Objects.hash(similarityFunction, vector1, vector2) (:89) and similarityFunction.name() + "(" + ... (:94).
5. The operands compare by value (ConstKnnFloatValueSource with Arrays.equals, FloatKnnVectorFieldSource by field name), so separately built EUCLIDEAN and COSINE sources over "field v, vector [1, 2]" are equal.
6. The wrappers pass it on. FunctionQuery.equals is sameClassAs(other) && func.equals(...) (FunctionQuery.java:180-182); FunctionRangeQuery compares its bounds and Objects.equals(valueSource, other.valueSource) (FunctionRangeQuery.java:115-125); FunctionScoreQuery compares its source (FunctionScoreQuery.java:144-149), and ValueSource.asDoubleValuesSource() wraps with Objects.equals(in, that.in) (ValueSource.java:249-253). FunctionScoreQuery use became possible in 10.4.0 (#15328).

One document with v = [2, 1] and the query vector [1, 2]: EUCLIDEAN gives 1 / (1 + 2) = 0.333, COSINE (1 + 0.8) / 2 = 0.9. equals is true and the hash codes are 1652613436 and 613814086 (the function's part of the hash is the enum constant's identity hash, so the numbers can change between JVM runs: both runs of this probe printed this pair, the probe behind the diagram printed -1860021253 and -1987185915; they differed in every recorded run). List.of(euclideanQuery).contains(cosineQuery) is true, a HashSet of the same query says false. FunctionRangeQuery(source, 0.5f, 1f, true, true) matches 0 documents with EUCLIDEAN and 1 with COSINE, and the two range queries are equals.

• Who is exposed. Callers that compare these value sources or the queries around them with equals alone, for example List.contains, indexOf or remove, an equality-based cache or request de-duplicator, or a test that asserts two queries differ, and that see two functions over the same operands.
• What does not break inside Lucene. The weights are never cached: isCacheable returns false in FunctionQuery (FunctionQuery.java:82-83), FunctionRangeQuery (FunctionRangeQuery.java:177-178) and the asLongValuesSource and asDoubleValuesSource wrappers (ValueSource.java:150-151, :230-231), so LRUQueryCache never keys on them. FunctionScoreQuery's scoring weight is cacheable only when both its inner weight and its source are (FunctionScoreQuery.java:280-282), so never with these sources; when scores are not needed it returns the inner query's weight itself (:117-120), which may be cached but whose matches do not depend on the function. BooleanQuery keeps its clauses in a Multiset / HashSet (BooleanQuery.java:132-140) and DisjunctionMaxQuery its disjuncts in a Multiset (DisjunctionMaxQuery.java:47); these compare hash codes first, which differ. Executed: a BooleanQuery with SHOULD EUCLIDEAN + SHOULD COSINE keeps both clauses (score 0.333 + 0.9 = 1.2333). No full sweep of in-tree consumers was run. One theoretical exception: with equal operands the two sources' hash codes are equal exactly when the two enum constants' identity hash codes are (Objects.hash multiplies the function's hash by 961, an odd number), and identity hash codes are not unique; in a JVM run where they collided, these sets and BooleanQuery.deduplicateClauses (BooleanQuery.java:714-716) would reach the wrong equals and could merge the two clauses. Never observed; not claimed.
• Downstream. Solr's vectorSimilarity() builds these classes, and its 4-argument form takes the function from the request. Whether a Solr cache could mix them up was not tested (the devil's-advocate verifier only re-did Solr's key hash arithmetic by hand), so it is not claimed.
• Default vs opt-in. Opt-in lucene/queries classes, not on the default search path. No index, codec or setting matters; assertions on or off behave the same.
• How likely. Rare: it needs an equals-only caller that meets two functions over the same operands, and no report of it was found since 9.7.0 (June 2023).

The test is byte-identical to the hunt's minimal repro and passes :lucene:queries:check -x test. It runs unchanged on the 10.6.0 and 9.7.0 release jars; 9.6.0 has none of these classes.

Executed: gradle on main e1faeba08bde with assertions on (seed 5EED1079) and off (5EED107A), both fail; the standalone runner on the 10.6.0 release jars with assertions on and off and on 9.7.0, all fail; the hunt's runs on main fail the same way. With the one-condition fix the test passes. JUnit's message prints only the second argument.

Use the same test as the regression test. It checks only the equals expectation: FunctionRangeQuery is exercised only by the illustration probe, and FunctionScoreQuery, the Float16 subclass, the query cache and downstream callers are not exercised.

cp $REPRO/TestVectorSimilarityFunctionEquals.java $LUCENE/lucene/queries/src/test/org/apache/lucene/queries/function/
cd $LUCENE && ./gradlew :lucene:queries:test --tests TestVectorSimilarityFunctionEquals -Ptests.seed=5EED1079 -Ptests.asserts=true
# also with -Ptests.asserts=false; nothing depends on the seed

TestVectorSimilarityFunctionEquals > testSimilarityFunctionIsPartOfEquals FAILED
    java.lang.AssertionError: Values should be different. Actual: COSINE(FloatKnnVectorFieldSource(v), ConstKnnFloatValueSource([1.0, 2.0]))
        at junit@4.13.2/org.junit.Assert.fail(Assert.java:89)
        at junit@4.13.2/org.junit.Assert.failEquals(Assert.java:187)
        at junit@4.13.2/org.junit.Assert.assertNotEquals(Assert.java:163)
        at junit@4.13.2/org.junit.Assert.assertNotEquals(Assert.java:177)
        at org.apache.lucene.queries.function.TestVectorSimilarityFunctionEquals.testSimilarityFunctionIsPartOfEquals(TestVectorSimilarityFunctionEquals.java:31)
1 test completed, 1 failed

# the same with -Ptests.asserts=false, and on the 10.6.0 (assertions on and off) and 9.7.0 release jars;
# with the fix: 1 test completed, 0 failed

• Contract. ValueSource declares equals and hashCode abstract (ValueSource.java:51-55); core Query.java:95-100 asks subclasses to implement equivalence "properly ... so that QueryCache works properly". java.lang.Object#hashCode requires objects that are equal by equals to return the same hash code; here they are equal and their hash codes differ.
• Maintainer rule. On LUCENE-7736 (#8787) jpountz wrote: "I think it's important for equals to behave correctly and never return true for values sources that could return different values." romseygeek agreed in the next comment ("I'll change things back again").
• Siblings. Other value sources put every value-deciding parameter in equals, for example LinearFloatFunction compares slope, intercept and source (LinearFloatFunction.java:85-91).
• Tests. TestKnnVectorSimilarityFunctions has no equals, hashCode or QueryUtils assertions; it passes unchanged with the fix.
• History. In PR #12253, commit 901febe400 (2023-06-13) compared similarityFunction in equals. uschindler's review asked for a getClass() check and a this == other / null check; commit 7219987d52 ("Addressing review") added them, and the function term disappeared, while hashCode kept the field. No review or conversation comment on the PR mentions leaving it out. The PR was squash-merged as 14c18d86245 (2023-06-14, 9.7.0).

The strongest "intended" case (built by the devil's-advocate verifier; the cross-model lenses rated it weak): equality identifies only the subclass and its two operands, the function is "implied" by the vector field, and Query.equals' javadoc speaks of "document-filtering properties", which a FunctionQuery (it matches every document) does not change.

Why it fails: it cannot explain why hashCode() and description() include the function, and equal objects with different hash codes are wrong under any equality definition. The function is an independent constructor argument (func never reads the field's own similarity, and Solr's 4-argument form lets a request choose it). For FunctionRangeQuery the function does decide which documents match (0 vs 1 hits in the probe, still equals). And the PR history is consistent with an accidental omission. Verdict: a defect, Low.

One fix was built and validated in a scratch worktree of e1faeba08bde; the alternative is listed with its trade-off.

In VectorSimilarityFunction.equals, compare similarityFunction too, so equals uses the same three fields as hashCode. The field is an enum, so == is exact (and null-safe: the constructor does not null-check the function, VectorSimilarityFunction.java:33-41, though a null one already fails in func and description()). Sources with the same function and operands stay equal, and the three subclasses and every wrapper are fixed at once. A reviewer may prefer casting once (VectorSimilarityFunction other = (VectorSimilarityFunction) o;); same behaviour.

--- a/lucene/queries/src/java/org/apache/lucene/queries/function/valuesource/VectorSimilarityFunction.java
+++ b/lucene/queries/src/java/org/apache/lucene/queries/function/valuesource/VectorSimilarityFunction.java
@@ -80,7 +80,8 @@ public abstract class VectorSimilarityFunction extends ValueSource {
   public boolean equals(Object o) {
     if (this == o) return true;
     if (o == null || getClass() != o.getClass()) return false;
-    return Objects.equals(vector1, ((VectorSimilarityFunction) o).vector1)
+    return similarityFunction == ((VectorSimilarityFunction) o).similarityFunction
+        && Objects.equals(vector1, ((VectorSimilarityFunction) o).vector1)
         && Objects.equals(vector2, ((VectorSimilarityFunction) o).vector2);
   }

Trade-offs

A deliberate behaviour change for callers that compare two such sources (or the queries around them) with different functions over the same operands: equals now tells them apart, which is the intended correction. Pairs with the same function and operands stay equal, with the same hash code.

What the checks validated

• Gradle, JDK 25, scratch worktree of e1faeba08bde: the test fails 1 of 1 on stock with assertions on and off; with the fix, the test plus TestKnnVectorSimilarityFunctions pass (2 suites, 12 tests, 0 failures) with assertions on and off (one upstream test skips itself without assertions); the whole :lucene:queries:test with assertions on passes (52 suites, 513 tests, 5 skipped, 0 failures); :lucene:queries:check -x test passes (checkGoogleJavaFormat, ecjLint, forbiddenApis).
• The hunt: the repro passes with the fix shadowed, and so do the independent reproducer's own test (Byte subclass inside FunctionQuery, with its identical fix) and the intended-lens reviewer's contract test.

Make hashCode match the current equals.

Trade-offs

Restores equals/hashCode consistency but makes the false equality permanent: sources that compute different values stay equal, against the LUCENE-7736 rule. Not validated.

Method: semi-formal (an automated bytecode field-set scan, an equality-congruence harness, then executed JUnit tests).

• A hunt unit (query-object contracts), 2026-10-26: a bytecode scan of the stock jars reported VectorSimilarityFunction.similarityFunction as read by hashCode, description() and each subclass's func but not by equals, for the Float, Byte and Float16 subclasses. The unit's equality harness witnessed it: a FunctionQuery over EUCLIDEAN equals the one over DOT_PRODUCT while their hash codes and scores differ.
• A later hunt unit (2026-11-02) re-ran the lead on main e1faeba08bde (unchanged) and reduced it to one public-API JUnit test, which fails with assertions on and off and passes with the one-condition fix.
• Independent verifiers, cross-model reviews, an adjudication and an audit then re-ran it (see the checks below).