Monitor scores a sub-query that occurs twice in a stored query only once

The Monitor (lucene-monitor) matches documents against stored queries. To select candidate queries quickly, its default QueryDecomposer splits a registered query into parts and collects them in a Set, which keeps equal queries once. When several parts of one query match a document, ScoringMatch and ExplainingMatch merge them by adding their scores. A BooleanQuery, however, counts a repeated clause at every occurrence. So when an equal sub-query occurs more than once among a registered query's decomposed SHOULD parts, or as a repeated required clause that is the only distinct required clause, with no SHOULD clause beside it, the Monitor reports, on every document on which that part matches, a score that counts the part once: (red apple) (green apple) on "red green apple" gets 0.39229372 instead of 0.5230583, and apple apple and +apple +apple get 0.13076457 instead of 0.26152915. The classic QueryParser builds all three from plain input. For these shapes, which queries match is unchanged. It is in every release since the module was imported (LUCENE-8766, 8.2.0). Keeping such a query whole in the decomposer (+25/-4 lines, one decomposer test added) fixes it, and the whole monitor module passes with that change.

• Monitor (lucene-monitor, org.apache.lucene.monitor): stores queries and matches incoming documents against them. A matcher chosen per match call decides what is reported: SIMPLE_MATCHER (query ids), ScoringMatch (ids and scores), ExplainingMatch (ids and an Explanation), HighlightsMatch (matching terms).
• QueryDecomposer: splits a registered query into parts that are indexed and run separately, so the presearcher can select only the parts whose terms occur in the document. Each part keeps the registered query's id. decompose returns a Set<Query>.
• Repeated part: a sub-query that occurs more than once, compared with Query.equals: apple in (red apple) (green apple), a repeated term apple apple, or a repeated required clause +apple +apple.

1. QueryDecomposer.decompose returns a Set<Query> (QueryDecomposer.java:42). decomposeBoolean adds the parts of every SHOULD clause to one LinkedHashSet (:80, :89) and the MUST/FILTER clauses to a HashSet (:82, :85-86); the DisjunctionMaxQuery branch adds its disjuncts' parts to a LinkedHashSet (:46-52). Equal parts are kept once.
2. For +apple +apple the HashSet holds one clause, so the keep-whole rule for more than one required clause (:94-95) does not fire, and the single required clause is decomposed (:99-101).
3. QueryCacheEntry.decompose stores one entry per Set element under the parent's id (QueryCacheEntry.java:51-59), and the Monitor runs every part the presearcher selects on its own (Monitor.java:299).
4. CandidateMatcher.addMatch merges two matches of one id with resolve() (CandidateMatcher.java:75-85). ScoringMatch.resolve returns match1.getScore() + match2.getScore() (ScoringMatch.java:42-44); ExplainingMatch.resolve builds a "sum of:" explanation (ExplainingMatch.java:44-52).
5. BooleanQuery.rewrite merges repeated SHOULD and MUST clauses "into a single clause with a combined boost" (BooleanQuery.java:438-455, :706-713): an unboosted repeat gets similarity.computeQueryTermWeight(count) (:743-746), which is count for default BM25 and the base Similarity. So the query counts the repeat, and with default BM25 the Monitor's score is too low by the score of each extra occurrence of a part that matches (with BM25's optional k3 > 0 the loss is smaller, and with k3 = 0 there is none).

A plain Boolean disjunction whose parts are all distinct is scored right (a DisjunctionMaxQuery with tie below 1 and a FILTER-only required clause are mis-scored even without a repeat: those are two separate sibling bugs), and so is a repeated part that does not match the document (it scores 0 both ways). A repeated required clause next to another required clause or a SHOULD clause is kept whole by line 94 and scored right: +apple +apple +red and +apple +apple green both get 0.39229372 from IndexSearcher and from the Monitor, while nested (+apple +apple) red is mis-scored (0.26152915 instead of 0.39229372). For these shapes matching is unchanged: a collapsed part matches exactly where its copies do. The one exception found is +apple +apple with minimumNumberShouldMatch=1, which the collapse turns into the shape of a separately registered minimumShouldMatch bug, so the Monitor reports a match that IndexSearcher does not.

Document {body: "red green apple"}, BM25, IndexSearcher on a one-document index vs ScoringMatch with BM25. Each of red, green and apple alone scores s = 0.13076457 (≈ idf 0.2876821 x tf 0.45454544; BM25 computes it as a float expression, so the decimal product differs in the last digits).

ExplainingMatch gives 0.3922937214374542 = "sum of:" green and "sum of:" (apple, red); ParallelMatcher and PartitionMatcher also report 0.39229372. IndexSearcher explains apple with "2.0 = boost" and rewrites the query to (body:apple)^2.0 body:green body:red. With ClassicSimilarity the shared-term case gives 1.5 instead of 2.0, and a a a under BM25 gives 0.13076457 instead of 0.39229375 (the whole query folds the weight 3 into BM25's float expression and rounds to 0.39229375; three summed parts give 0.39229372).

• Who. lucene-monitor users with the default QueryDecomposer (MonitorConfiguration.java:38) who read scores through ScoringMatch (directly or through ParallelMatcher / PartitionMatcher, which report the same values) or ExplainingMatch, and register a query in which an equal sub-query occurs more than once among its decomposed SHOULD parts (a repeated word in user input, apple apple; overlapping expansions that share a term, (red apple) (green apple)), or a repeated required clause that is the only distinct required clause of its BooleanQuery, with no SHOULD clause beside it (+apple +apple, or nested, (+apple +apple) red). +apple +apple +red and +apple +apple green are kept whole and scored right.
• Default vs opt-in. The Monitor configuration and the decomposer are the defaults; lucene-monitor is an optional module, and the matcher is chosen per match call (no default); ScoringMatch and ExplainingMatch are two of the four documented matchers. The classic QueryParser builds all three shapes from plain strings.
• Scale. None needed: one query and one document show it, deterministically, with assertions on and off.
• Effect. Too-low scores, and explanations that omit the repeat, which can mis-rank or mis-threshold stored queries against each other. For these shapes the set of matching queries is right (the one exception found, +apple +apple with minimumNumberShouldMatch=1, is described in the bug section); nothing failed or was corrupted in any run, and the values look plausible.
• How likely. Rare: it needs the scoring matchers and a stored query with a repeated part, which is uncommon within one query; the one public Monitor integration found (an open Solr pull request) defers the scoring matchers. The lack of reports since 2019 is weak evidence, because the wrong values look reasonable.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<0.5230583> but was:<0.39229372>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:577)
       org.junit.Assert.assertEquals(Assert.java:701)
       org.apache.lucene.monitor.TestMonitorSharedSubqueryScore.testSharedTermIsScoredAtEachOccurrence(TestMonitorSharedSubqueryScore.java:57)

• Monitor contract. package-info.java:40-45: ScoringMatch.matchWithSimilarity "returns the set of matching queries, with the score that each one records against a Document"; ExplainingMatch is "similar to ScoringMatch, but include the full Explanation". The natural reading is the query's own score.
• Core contract. BooleanQuery.deduplicateClauses merges repeated queries "into a single clause with a combined boost" (BooleanQuery.java:706-713); core has counted repeated scoring clauses since LUCENE-7925 (#8974, 2017, "Collapse duplicate SHOULD or MUST clauses by summing up their boosts"), asserted by TestBooleanRewrites.testDeduplicateShouldClauses and testDeduplicateMustClauses. The classic QueryParser's apple apple is explained with "2.0 = boost" and rewritten to (body:apple)^2.0.
• No distinct-parts semantic is documented. QueryDecomposer's javadoc says only "Split a disjunction query into its constituent parts, so that they can be indexed and run separately"; CandidateMatcher.resolve says "combine them". No CHANGES or MIGRATE entry discusses it.
• The only test that pins a score value. ConcurrentMatcherTestBase.testMatchesAreDisambiguated says "We need to check that each query ends up with the sum of the scores for the 'test' and 'doc' parts" and asserts 1.4874471; an executed check showed that this is IndexSearcher's score for the whole query. The sum was meant to rebuild the query's own score, which it does for a plain Boolean disjunction whose parts are all distinct (the DisjunctionMaxQuery and FILTER sibling bugs aside). The test does not literally state equality with the whole query's score, and no upstream test has a query with a repeated part.
• History. The predecessor library (Luwak) kept the parts in Lists and its resolve kept the larger match, so a repeat could not change a score there. The 2019 import (LUCENE-8766) changed the parts to a Set and resolve to a sum in the same commit, and rewrote the shared test from "the score for the 'test' subquery, not the 'doc' subquery" to "the sum of the scores", with no recorded discussion. #14886 / #14885 (2025) discussed the Set only for its iteration order.

The strongest works-as-designed case (the devil's advocate and the cross-model reviewer): decompose has returned Set<Query> in the public API since 8.2.0, so the parts are defined by uniqueness, and running an equal part twice would be wasted work; nothing literally promises that a Monitor score equals IndexSearcher's; the decomposer already misreports DisjunctionMaxQuery and FILTER scores, and Luwak's max never equalled a disjunction's score either, so Monitor scores could be called a heuristic; and a decomposer that never splits is a supported customization.

Why it fails: nothing documents a distinct-parts semantic, the package javadoc reads as the query's own score, and the port's own test treats the sum of the parts as the query's score. The Set is fine for matching; the defect is that the summing resolve assumes each part occurs once. Core deliberately keeps the weight of a repeat, and the Monitor contradicts it on queries built from plain QueryParser input. The other mis-scorings are separate bugs, and a workaround is not a contract. Verdict: not intended; a defect since the import. At worst upstream answers with a javadoc change.

One fix was built and run in a scratch worktree at a9779ef3c193; it is the finder's fix, cleaned up and with a decomposer test, and is recommended. A broader change from the DisjunctionMaxQuery sibling's doc, which runs the registered query whole, also fixes this bug.

decomposeBoolean counts the SHOULD parts and the required clauses before they go into the Sets, and returns the query as a singleton when a Set is smaller than its count; the DisjunctionMaxQuery branch does the same for its disjuncts' parts. The new TestQueryDecomposer.testRepeatedPartsAreNotDecomposed asserts that hello hello, +hello +hello, (hello world) (goodbye world) and DisMax(f:t1, f:t1) stay whole. +25/-4 lines in QueryDecomposer.java, +12 in the test. The snippet below is condensed (the source spells out BooleanClause.Occur).

int shouldParts = 0, mandatoryClauses = 0;
for (BooleanClause clause : q) {
  if (clause.occur() == MUST || clause.occur() == FILTER) {
    mandatory.add(clause.query());
    mandatoryClauses++;
  } else if (clause.occur() == MUST_NOT) exclusions.add(clause.query());
  else {
    Set<Query> decomposed = decompose(clause.query());
    shouldParts += decomposed.size();
    subqueries.addAll(decomposed);
  }
}
if (subqueries.size() < shouldParts || mandatory.size() < mandatoryClauses)
  return Collections.singleton(q);

Trade-offs

• A query with a repeated part becomes one presearcher entry instead of one per distinct part, and runs whole whenever it is selected. The cost was not benchmarked. Decomposition happens at register, before any matcher is chosen, so the change applies to every matcher, SIMPLE_MATCHER and highlighting included: for such queries the cache entries, the presearcher's selectivity and getQueriesRun() change too. Matching is unchanged for the three repro shapes (the module's matching and presearcher tests pass, and a match-set probe agrees on every row with the fix). The public counts Monitor.getDisjunctCount() and QueryCacheStats drop for a query that had several distinct parts on stock, such as (red apple) (green apple) (3 entries become 1); apple apple, +apple +apple and DisMax(a, a) keep one entry.
• Queries whose parts are all distinct decompose exactly as before (the 11 existing decomposer tests pass).
• The rule is conservative: it also keeps whole some repeats whose score the collapse does not change, such as DisMax(a, a) with tie 0 or any repeat under BM25 with k3 = 0. No such case was compared.
• Side effect: +apple +apple with minimumNumberShouldMatch=1, where the collapse made the Monitor report a match that IndexSearcher does not, is now kept whole and agrees. The separately registered minimumShouldMatch bug (one distinct required clause) needs its own fix.
• The DisjunctionMaxQuery sibling's fix edits the same DisjunctionMaxQuery branch, so the two patches conflict textually but combine logically; they were not run together.

What the checks validated

Gradle in a scratch worktree at a9779ef3c193, seed DEADBEEF: the whole :lucene:monitor:test with the 3 repro classes added, 35 classes, 197 tests, 0 failures, 1 skipped (a @Nightly test), with assertions on and off; all 12 repro tests that fail on stock pass. :lucene:monitor:check -x test with the fix and the primary test (google-java-format, ecj lint, forbidden APIs): BUILD SUCCESSFUL. The finder's original diff: 196 tests, 0 failures. The match-set probe with the fix: 15 of 15 rows agree.

From the DisjunctionMaxQuery sibling's doc: each QueryCacheEntry also keeps its parent MonitorQuery's query, and the Monitor runs that whole query once per query id when the presearcher selects any of its parts. +17/-3 lines in QueryCacheEntry.java and Monitor.java; written as a sketch.

Trade-offs

• Fixes this bug and both sibling decomposition bugs (DisjunctionMaxQuery summed, FILTER clause scored).
• It changes every matcher, not only the scoring ones; getQueriesRun() no longer equals the number of queries executed; a large disjunction runs whole when one part is selected; every cache entry keeps a reference to its parent's query. Not benchmarked; a design change for maintainers to weigh.

What the checks validated

In the sibling's validation, with assertions on: this bug's finder test fails 3 of 4 on stock main and passes 4 of 4 with the change; the whole module with the sibling repros, 37 classes, 208 tests, 0 failures, 1 skipped. Not run with assertions off or through check.

Keep every occurrence as its own part, so the sum counts each one.

Trade-offs

An API change: decompose, decomposeBoolean and decomposeBoostQuery are public and overridable. It would also change the per-part cache ids that #14885 made stable across JVMs. Not run.

Wrap a part that occurs n times in a BoostQuery of n.

Trade-offs

Exact only for linear similarities: on main an unboosted repeat is weighted by Similarity.computeQueryTermWeight(n), which saturates for BM25 with k3 >= 0, and the decomposer does not know the Similarity, which the matcher chooses per match call. Not run.

A note in the package javadoc and on ScoringMatch / ExplainingMatch.

Trade-offs

A legitimate maintainer choice, but queries built from plain QueryParser input stay mis-scored. Not run.

A QueryDecomposer subclass whose decompose returns the query as a singleton (for every query, or only when a part repeats), set with new MonitorConfiguration().setQueryDecomposer(...). No Lucene change.

Trade-offs

Gives IndexSearcher's exact score (the independent reproducer's oracle, the devil's-advocate probe and this doc's control test); every application has to know about it.

Method: bounded-exhaustive translation validation against a reference semantics (the hunt's "semi-formal" tier).

• The finder's harness ran six matchers (simple, scoring, explaining, highlighting, parallel, partition) on the stock decomposer, and three of them on six decomposer variants, against IndexSearcher on a one-document index: every depth-1 query (420) on 55 documents, every depth-2 query (7,685) on 19 documents, and 3,000 random depth-3 queries on 34 documents; 6,506,760 checks with assertions on and the same with them off.
• First signal (depth 1): body:a body:a on the document "a", 0.13076457 instead of 0.26152915. The unit found it itself; the planning probe for this area had not.
• Root attribution with public decomposer subclasses: the variant that keeps a query whole when its parts would collapse was the only single change that removed 252, 1,142 and 365 score-mismatch rows in the three tiers, for the scoring and the explaining matcher each; further rows were removed by it together with a sibling variant. The DisjunctionMaxQuery and FILTER variants explain the two sibling bugs and leave this repro failing.
• A Z3 model of the score algebra over the reals confirmed that a collapsed repeat keeps the score only when the repeated part does not match.
• The finder then wrote the public-API JUnit test, and independent checkers reproduced it.