TermInSetQuery.equals ignores the rewrite method, so set queries that match different documents compare equal and IndexWriter drops one of two such deletes

Since 9.6.0 TermInSetQuery is a MultiTermQuery, and its rewrite method decides how it runs: on postings (the default), on doc values (DOC_VALUES_REWRITE, which SortedSetDocValuesField.newSlowSetQuery returns), or keeping only the top N terms. MultiTermQuery#equals and hashCode compare the rewrite method, but TermInSetQuery overrides both with a 2017 version that compares only the packed field and terms. So queries that match different documents compare equal: on a field with only doc values, new TermInSetQuery("f", [a]) and newSlowSetQuery("f", [a]) match 0 and 1 documents; on an ordinary field, new TermInSetQuery("f", [a, b]) and its TopTermsScoringBooleanQueryRewrite(1) variant match 2 and 1. Anything keyed on the un-rewritten query treats each pair as one. The main case inside Lucene is IndexWriter's delete-by-query buffer, a HashMap<Query, Integer> that keeps the first of two equal keys: with the default IndexWriterConfig, deleteDocuments with both doc-values-case queries leaves the document live, and in the other order a document added between two calls is deleted although no call matched it. The monitor module's QueryDecomposer also merges such a pair of disjuncts, so with presearcher filtering off a doc-values-only match is missed. Lucene's search path is not affected, because it compares rewritten queries. Affected: 9.6.0 to 10.6.0 and main; the fix is to delegate to MultiTermQuery#equals/hashCode, as the other core subclasses do.

1. The rewrite method decides what matches. TermInSetQuery extends MultiTermQuery (TermInSetQuery.java:73) and has a public TermInSetQuery(RewriteMethod, String, Collection<BytesRef>) constructor (:87-92); the class javadoc invites custom rewrite methods "to define different execution behavior, such as relying on doc values ... or if scores are required" (:65-69). MultiTermQuery#rewrite is final: return rewriteMethod.rewrite(indexSearcher, this); (MultiTermQuery.java:325-327).
2. The parent compares it, the subclass does not. MultiTermQuery#equals/hashCode include rewriteMethod and field (MultiTermQuery.java:336-352). TermInSetQuery overrides both: equals is sameClassAs(other) && equalsTo(...), equalsTo compares termDataHashCode and termData only, and hashCode is 31 * classHash() + termDataHashCode (TermInSetQuery.java:203-217). It never calls the parent. Every other concrete MultiTermQuery outside sandbox (AutomatonQuery, so WildcardQuery and RegexpQuery, PrefixQuery, FuzzyQuery, TermRangeQuery, join TermsQuery) calls super.equals, except MultiFacetQuery (facet module), which extends TermInSetQuery and inherits its override. All of them but join TermsQuery also build hashCode on super.hashCode() (join TermsQuery.java:127-129 uses classHash()).
3. The doc-values factory is such a query. SortedSetDocValuesField.newSlowSetQuery returns new TermInSetQuery(MultiTermQuery.DOC_VALUES_REWRITE, field, values) (SortedSetDocValuesField.java:130-132; the same in SortedDocValuesField.java:126-128). TermInSetQuery.newIndexOrDocValuesQuery (:104-110), used by KeywordField.newSetQuery, pairs a postings and a doc-values TermInSetQuery that compare equal.
4. IndexWriter keys buffered deletes on the query. deleteDocuments(Query...) (IndexWriter.java:1883-1898) reaches the delete queue, whose QueryArrayNode.apply calls bufferedUpdates.addQuery(query, docIDUpto) for each query (DocumentsWriterDeleteQueue.java:462-472). BufferedUpdates.deleteQueries is a HashMap<Query, Integer> (BufferedUpdates.java:64); addQuery does deleteQueries.put(query, docIDUpto) (:110-116), which for an equal key keeps the first key object and replaces only its doc-id limit.
5. Only the first query runs. FrozenBufferedUpdates copies the keys (FrozenBufferedUpdates.java:93-99) and runs each once per segment (:388-420). The postings query matches nothing on a field without postings, so the document stays live. In the other order, the stored doc-values query takes the later limit and also deletes a document added between the two calls.
6. Search is not affected. BooleanQuery rewrites its clauses before it removes duplicate FILTER and MUST_NOT clauses (BooleanQuery.java:299-362), LRUQueryCache keys on the rewritten query (LRUQueryCache.java:834), and IndexOrDocValuesQuery.rewrite compares rewritten halves (IndexOrDocValuesQuery.java:131). In the executed cases the rewritten forms differ (different wrapper classes), so those keys stay distinct.

• What goes wrong. A delete that silently does not take effect, so the document stays searchable; or, in the other order, a document that no call matched is deleted. Both are committed. No error is raised.
• Configuration. The default IndexWriterConfig, codec and NRT reader suffice. One call with both queries always collides. Two calls collide only if nothing applies the buffered deletes in between: a flush, an NRT reopen or a commit does, and so can IndexWriter itself when buffered deletes use too much heap (DocumentsWriter.java:153-166). A commit between the calls avoids it.
• What it takes. Two TermInSetQuery objects with the same field and terms whose rewrite methods select different documents. With the documented pairing (postings TermInSetQuery next to newSlowSetQuery) that needs a field that lacks postings or doc values on the documents concerned, for example a doc-values-only field or documents indexed before a schema change. With a top-terms or scoring rewrite an ordinary field is enough.
• Other consumers. An application cache or set keyed on un-rewritten Query objects serves one query's result for the other (an independent reproducer: a map returned 2 for the top-1 query, whose true count is 1). Wrapper equality inherits it: a BooleanQuery with a FILTER clause of either variant compares equal. The monitor module's QueryDecomposer keeps the un-rewritten clauses of a registered query in sets (QueryDecomposer.java:77-102): a BooleanQuery with both variants as SHOULD clauses decomposes into 1 part, not 2, and with Presearcher.NO_FILTERING a document whose field has only the doc value gets 0 matches where a direct search counts 1 (executed on main; with the fix 2 parts and 1 match). With the default presearcher both builds give 0, because it does not select doc-values-only fields.
• Not affected, of the consumers checked. Lucene's search path: BooleanQuery de-duplication, LRUQueryCache and IndexOrDocValuesQuery compare rewritten queries, which differ in the executed cases (impact probes on main). MultiFacetQuery inherits the override but always uses the default rewrite method. Other in-tree code that keeps un-rewritten queries in sets or maps was not audited.
• Likelihood. Contrived for the delete path: no caller doing this is known, and the recorded searches found no report since 9.6.0 (2023). Index size does not matter; the repro has one document.

The class sits in org.apache.lucene.search, next to TestTermInSetQuery, and uses nothing package-private. The test framework's random directory and codec do not matter: 20 of 20 derived seeds fail the same way. It compiles on 9.10.0 and later (the newSlowSetQuery(String, Collection) overload); on 9.6.0 to 9.9.x use the BytesRef... overloads instead.

Regression test: this same test, as written. With the fix it passes, together with the core format, ECJ and forbidden-apis checks.

cp $REPRO/TestTermInSetQueryDeleteByDocValuesSetQuery.java $LUCENE/lucene/core/src/test/org/apache/lucene/search/
cd $LUCENE && ./gradlew :lucene:core:test --tests org.apache.lucene.search.TestTermInSetQueryDeleteByDocValuesSetQuery -Ptests.seed=DEADBEEF
# assertions off fails the same way: add -Ptests.asserts=false (the recorded runs added --offline --no-daemon)

# gradle, scratch worktree of 645d4fe14a20, seed DEADBEEF, assertions on
TestTermInSetQueryDeleteByDocValuesSetQuery > testDeleteByPostingsAndDocValuesSetQuery FAILED
    java.lang.AssertionError: live docs after deleteDocuments(q1, q2) expected:<0> but was:<1>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:81CC3BFBF12E5807]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.apache.lucene.search.TestTermInSetQueryDeleteByDocValuesSetQuery.testDeleteByPostingsAndDocValuesSetQuery(TestTermInSetQueryDeleteByDocValuesSetQuery.java:42)

# standalone runner, seed DEADBEEF: 1 of 1 failed, same message, on
main 645d4fe14a20 (-ea and -da), 10.6.0, 10.5.2, 10.0.0, 9.12.3, 9.10.0

# with the recommended fix: the test passes (assertions on and off)

• The delete contract. IndexWriter.deleteDocuments(Query...): "Deletes the document(s) matching any of the provided queries" (IndexWriter.java:1875). The repro asserts exactly this, so it needs no argument about what equals should return.
• The equality contract. Query#equals is "required so that QueryCache works properly", and "Typically a query will be equal to another only if it's an instance of the same class and its document-filtering properties are identical to those of the other instance" (Query.java:96-107). Here they differ: 0 vs 1 and 2 vs 1 matching documents.
• The parent and the siblings. MultiTermQuery#equals/hashCode include the rewrite method (MultiTermQuery.java:336-352), and every other concrete subclass outside sandbox calls super.equals, except MultiFacetQuery, which inherits TermInSetQuery's override (the core siblings also call super.hashCode; join TermsQuery does not). For BlendedTermQuery, a test requires two queries that differ only in their rewrite method to be unequal (TestBlendedTermQuery.java:39-49).
• History. The terms-only equals dates from LUCENE-7624 (2017), when TermInSetQuery extends Query had no rewrite method. #12156 (9.6.0) made it a MultiTermQuery with a public rewrite-method constructor and changed no line of equals/hashCode; its discussion never mentions equality. Before #12175 (9.6.0) the doc-values set query was a separate class that never compared equal.
• Maintainer statements and precedent. LUCENE-10431 (#11467) kept the rewrite method in MultiTermQuery#equals (9.x left it out of hashCode only, because it was mutable there). In LUCENE-7810 (#8861), a false-positive query equality fixed as a bug, Adrien Grand wrote, about taking the score mode into account in the join query's equals, that "two queries should only be equal if they matche [sic] the same docs and give them the same scores" (2017-05-16). Similar omissions were fixed as bugs: LUCENE-10026 (CombinedFieldQuery), LUCENE-10552 (KnnVectorQuery), #12260 (SynonymQuery).
• Nothing pins the current behaviour. TestTermInSetQuery's equality tests use only the default rewrite method (TestTermInSetQuery.java:315-376); no CHANGES or MIGRATE entry describes it.

The strongest case for "intended" (a devil's-advocate verifier and a cross-model review): the rewrite method is an execution strategy. On a field indexed with the same values in postings and doc values, the layout KeywordField and IndexOrDocValuesQuery assume, the default, doc-values and constant-score rewrites match the same documents, so treating them as equal is harmless and lets the two halves of newIndexOrDocValuesQuery compare equal. Query#equals says only "Typically"; Lucene's own consumers compare rewritten queries; and 9.x once took the rewrite method out of MultiTermQuery#hashCode (LUCENE-10431).

Why it fails: TermInSetQuery's own javadoc offers rewrite methods that change matches and scores, not only execution (top-terms(1): 2 vs 1 documents; SCORING_BOOLEAN_REWRITE: top score 1.0 vs 0.315). A doc-values-only field is a legal use of newSlowSetQuery, and there the "same documents" premise fails (1 vs 0). The deleteDocuments contract breaks however one reads "Typically". LUCENE-10431 kept the rewrite method in equals and left it out of the 9.x hashCode only because it was mutable there; on main it is final and in both. The independent reproducer, the devil's-advocate verifier and the intended-lens review returned "real" (the reading-only refute review's "not real" was procedural: it could not run Java); the adjudication confirmed with high confidence. The practical reach is narrow, so a maintainer may still rate it low priority.

Both fixes that make equality include the rewrite method were validated through gradle in a scratch worktree of 645d4fe14a20; the recommended one is the idiomatic form. A change to the delete buffer instead was not built.

Call super.equals(other) instead of sameClassAs(other), and build hashCode on super.hashCode() instead of classHash(), as PrefixQuery, AutomatonQuery, FuzzyQuery and TermRangeQuery do. The parent checks the class, the field and the rewrite method.

@Override
public boolean equals(Object other) {
  // super.equals checks the class, the field and the rewrite method
  return super.equals(other) && equalsTo(getClass().cast(other));
}

@Override
public int hashCode() {
  return 31 * super.hashCode() + termDataHashCode;
}

Trade-offs

• Side effect. The two halves of TermInSetQuery.newIndexOrDocValuesQuery (and so of KeywordField.newSetQuery) stop comparing equal, which restores the pre-9.6 distinction between the doc-values and the postings set query. Nothing in-tree relies on their equality; IndexOrDocValuesQuery.rewrite compares rewritten halves, which already differ.
• Cost. One extra rewrite-method and field comparison per equals, before the term data. The built-in rewrite methods are singletons or value-equal, so hash codes stay stable. A custom RewriteMethod without equals makes queries built with different instances unequal (cache misses only), as for every other MultiTermQuery today.
• 9.x. There MultiTermQuery#hashCode deliberately leaves out the then-mutable rewrite method while equals keeps it (LUCENE-10431), so delegating gives the right result on 9.x without a special backport (read, not run on 9.x).

What the checks validated

• Gradle, scratch worktree of 645d4fe14a20, OpenJDK 25.0.3, seed DEADBEEF (2026-10-31). Stock: the repro fails. With the fix: it passes with assertions on and off.
• The repro plus the 9 closest suites (TestTermInSetQuery, TestIndexOrDocValuesQuery, TestMultiTermQueryRewrites, TestDocValuesRewriteMethod, TestMultiTermConstantScore, TestBlendedTermQuery, TestSortedSetDocValuesSetQuery, TestKeywordField, TestIndexWriterDelete): 73 tests, 8 skipped (nightly or monster), 0 failures. The nightly TestTermInSetQuery.testDuel and TestIndexOrDocValuesQuery with -Ptests.nightly=true: 20 tests, 0 failures.
• :lucene:core:test over org.apache.lucene.search.* and org.apache.lucene.document.*: 2,838 tests, 75 skipped, 0 failures; :lucene:monitor:test (181), :lucene:facet:test (306), :lucene:join:test (104): 0 failures.
• checkGoogleJavaFormat, ecjLintMain, ecjLintTest (with the repro test in place), forbiddenApisMain, forbiddenApisTest: pass.

Add rewriteMethod.equals(other.rewriteMethod) to equalsTo and rewriteMethod.hashCode() to hashCode, keeping the field implicit (it is encoded in the term data).

return termDataHashCode == other.termDataHashCode
    && rewriteMethod.equals(other.rewriteMethod)
    && termData.equals(other.termData);
...
return 31 * (31 * classHash() + termDataHashCode) + rewriteMethod.hashCode();

Trade-offs

Same effect on main, but it duplicates the parent's logic, and a 9.x backport must leave the rewrite method out of hashCode by hand.

What the checks validated

Gradle on 645d4fe14a20: the repro plus TestTermInSetQuery, TestIndexOrDocValuesQuery, TestKeywordField and TestSortedSetDocValuesSetQuery: 30 tests, 2 skipped, 0 failures. Earlier also checked by the finder, the independent reproducer, the adjudication and the repro minimization.

Make BufferedUpdates keep every delete query object (an identity map or a list).

Trade-offs

Fixes only the delete path, not application caches or wrapper equality, and loses the de-duplication of genuinely equal delete queries.

Method: semi-formal (a static field-set scan plus bounded-exhaustive testing, then public-API JUnit tests).

• A hunt unit on query object contracts scanned the bytecode of all 142 concrete Query classes in the main jars and listed, for each class, the fields that other methods, such as rewrite(), read but equals/hashCode do not. MultiTermQuery.rewriteMethod was read by rewrite() but not by TermInSetQuery.equals/hashCode.
• The unit's bounded-exhaustive equality-congruence harness ran small queries on small indexes and found equal TermInSetQuery pairs with different matching documents, different scores, and unequal rewritten forms. The finder reduced it to a JUnit on the public API.
• An independent reproducer, a devil's-advocate verifier (who added the doc-values-only case), cross-model reviews (OpenAI GPT-5.6, reading only) and an adjudication checked it. The security screen then executed the IndexWriter.deleteDocuments consequence, the follow-up re-ran it with the default configuration, and the repro was minimized to one test.