Monitor scores a lone FILTER clause, so FILTER-only queries get a positive or inflated score

The Monitor (lucene-monitor) matches documents against stored queries. To select candidate queries quickly, its default QueryDecomposer splits a registered query into parts; when several parts of one query match a document, ScoringMatch and ExplainingMatch add their scores. The decomposer groups FILTER clauses with MUST clauses, and when a BooleanQuery has exactly one required clause and no SHOULD clause that yields decomposed parts it replaces the query by the decomposition of that clause. For a MUST clause that is right. For a FILTER clause, which is "Like MUST except that these clauses do not participate in scoring", the FILTER flag is lost and the part scores: #body:apple gets 0.13076457 instead of 0, #body:apple -body:kiwi the same, #(body:apple body:banana) 0.26152915 instead of 0, and (#body:apple) body:banana 0.26152915 instead of 0.13076457. Core gives the same one-clause shape a score of 0 in BooleanQuery.rewrite. Which queries match is unchanged by this bug. The code came with the 2019 import of the Luwak library (LUCENE-8766) and is in every release from 8.2.0 on. Wrapping each decomposed part of a lone FILTER clause in a FILTER-only BooleanQuery fixes it, and the whole monitor module passes with that change.

• Monitor (lucene-monitor, org.apache.lucene.monitor): stores queries and matches incoming documents against them. A matcher chosen per match call decides what is reported: SIMPLE_MATCHER (query ids), ScoringMatch (ids and scores), ExplainingMatch (ids and an Explanation), HighlightsMatch (matching terms).
• QueryDecomposer: splits a registered query into parts that are indexed and run separately, so the presearcher can select only the parts whose terms occur in the document. Each part keeps the registered query's id.
• FILTER clause #q: a required clause that does not score ("Like MUST except that these clauses do not participate in scoring").

1. decomposeBoolean puts FILTER clauses into the same mandatory set as MUST clauses (QueryDecomposer.java:84-86).
2. With more than one distinct required clause, or one plus decomposed SHOULD parts, the query is kept whole (:93-95): those shapes are scored right.
3. With exactly one required clause and no decomposed SHOULD parts (normally: no SHOULD clause), it returns the decomposition of that clause, "decompose the MUST clause instead" (:97-101). For a FILTER clause the occur is lost: #body:apple is stored as the plain body:apple. With MUST_NOT clauses each part is rebuilt as +part -exclusion (:103-116), still scoring.
4. Each part runs on its own as the matcher's query (Monitor.java:299). ScoringMatch collects its score and ExplainingMatch explains it (ExplainingMatch.java:38); two matches of one id are merged by resolve, which adds the scores (CandidateMatcher.java:75-85, ScoringMatch.java:42-44, ExplainingMatch.java:44-53). So a FILTER-only clause nested next to a scoring clause is counted on top of it.

Core treats the same one-clause shape as non-scoring: BooleanQuery.rewrite turns a single FILTER clause into new BoostQuery(new ConstantScoreQuery(query), 0), "no scoring clauses, so return a score of 0" (BooleanQuery.java:289-291). The decomposer's rule is analogous to that rewrite (core's applies only to a query with exactly one clause and minimumNumberShouldMatch 0), but it drops exactly the FILTER case.

Document body "apple banana", BM25, IndexSearcher on a one-document index vs the default Monitor:

SIMPLE_MATCHER matches in every case (all with the default minimumNumberShouldMatch 0). A Monitor configured with a decomposer that never splits gives IndexSearcher's values.

• Who. lucene-monitor users with the default QueryDecomposer (MonitorConfiguration.java:38) who read scores through ScoringMatch (directly or through ParallelMatcher / PartitionMatcher, which report the same values) or ExplainingMatch, and register a BooleanQuery whose only required clause is a single FILTER clause with no SHOULD clause that yields decomposed parts (MUST_NOT clauses and boosts allowed): on its own, or nested as a SHOULD clause next to scoring clauses.
• Where such queries come from. Built in code (for example a "must match, do not score" condition emitted as a filter-only boolean), or by the XML query parser (occurs="filter"). The classic and standard query parsers do not produce FILTER clauses.
• Default vs opt-in. The Monitor configuration and the decomposer are the defaults; lucene-monitor is an optional module, and the matcher is chosen per match call. ScoringMatch and ExplainingMatch are two of the four documented matchers.
• Scale. None needed: one query and one document show it, deterministically, with assertions on and off.
• Effect. A part that should add nothing adds its full term score: FILTER-only queries get a positive score, nested ones are over-scored (here doubled), and ExplainingMatch shows a scoring explanation for a non-scoring clause. Applications that rank or threshold on Monitor scores see it. The set of matching queries is right; nothing fails.
• How likely. Rare: the shape is uncommon (FILTER next to a MUST clause, or next to a SHOULD clause that has decomposed parts, in the same BooleanQuery is not affected), it needs a scoring matcher, and the saved searches found no user report since 2019 (weak evidence, because the wrong values look plausible).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<0.13076457> but was:<0.26152915>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:577)
       org.junit.Assert.assertEquals(Assert.java:701)
       org.apache.lucene.monitor.TestMonitorFilterClauseScore.testFilterOnlyClauseInDisjunction(TestMonitorFilterClauseScore.java:53)

• Contract. BooleanClause.java:35: FILTER is "Like MUST except that these clauses do not participate in scoring". package-info.java:40-45: ScoringMatch.matchWithSimilarity "returns the set of matching queries, with the score that each one records against a Document"; ExplainingMatch is "similar to ScoringMatch, but include the full Explanation". No javadoc, CHANGES or MIGRATE entry says Monitor scores are approximate or that FILTER parts score.
• Design. The matcher layer rebuilds the whole query's score from its parts (CandidateMatcher.resolve: "If two matches from the same query are found (for example, two branches of a disjunction), combine them"). The only upstream test that pins a Monitor score value, ConcurrentMatcherTestBase.testMatchesAreDisambiguated, expects "the sum of the scores for the 'test' and 'doc' parts" to equal the whole query's score. In the tests the rebuild is exact for the control shapes (a single MUST clause, plain and boosted disjunctions of distinct parts, FILTER next to MUST or SHOULD, two distinct FILTER clauses); DisjunctionMaxQuery shapes and repeated sub-queries are not exact, and those are the sibling bugs LFV-852 and LFV-856.
• Core. BooleanQuery.rewrite handles the same one-clause FILTER shape as "no scoring clauses, so return a score of 0" (since 2015).
• Tests. TestQueryDecomposer.testFilterAndShouldClause asserts only that FILTER+SHOULD is not split; no upstream test asserts the score or explanation of a FILTER-only query.
• History. The MUST || FILTER line came from the Luwak library (PR #151, issue #150, 2017) and was imported unchanged in 2019. It fixed matching: FILTER clauses had been treated as SHOULD, so #a b was split. The review and tests never mention scores, and before that change a FILTER-only query was already split into the same scoring part, and nothing found in that history shows a decision that FILTER parts should score.
• Inconsistent today. The stock Monitor already scores #body:apple #body:banana (two distinct FILTER clauses, kept whole) 0, but #body:apple 0.13076457: whether a FILTER-only query scores depends on a decomposer internal.

The strongest works-as-designed case (the devil's-advocate check, the cross-model reviewer's reading and the adjudication's caveats): QueryDecomposer promises only a split ("Split a disjunction query into its constituent parts, so that they can be indexed and run separately"), the stored part is documented as "The (possibly partial due to decomposition) query", so ExplainingMatch explains what ran; grouping FILTER with MUST was deliberate; the module already changes scores elsewhere (the DisjunctionMaxQuery sibling); and ScoringMatch drops zero scores, so the stock behaviour is what makes a bare FILTER-only query appear in ScoringMatch at all. A maintainer could call it a documentation gap and point to a custom decomposer.

Why it fails: the deliberate part was about matching, and keeping the parts non-scoring keeps the same split and the same matches (the fix passes the whole module, presearcher and highlighting tests included). ExplainingMatch has no zero-score rule and contradicts IndexSearcher.explain; in the nested form the query is reported either way and its score is doubled; neither depends on the zero-score drop. "Partial query" describes the split, not a licence to change a clause's occur. Verdict: not intended; a defect since the import. Say up front that after a fix a bare FILTER-only query scores 0 and therefore no longer appears in ScoringMatch results, as two-FILTER queries already do not.

Two fixes were built and run in a scratch worktree at a9779ef3c193; both pass the whole monitor module with the repro classes. Fix A keeps the split; fix B is smaller but keeps such queries whole.

The finder's change, unchanged, plus TestQueryDecomposer.testSingleFilterClauseIsDecomposedIntoFilterParts, which asserts that #(hello world) becomes #hello, #world and that #hello -goodbye becomes +(#hello) -goodbye. +11/-1 in QueryDecomposer.java, +24 in the test.

Set<Query> mandatory = new HashSet<>();
Set<Query> scoringMandatory = new HashSet<>();
for (BooleanClause clause : q) {
  if (clause.occur() == BooleanClause.Occur.MUST) scoringMandatory.add(clause.query());
  ... // unchanged
}
...
if (mandatory.size() == 1) {
  Query required = mandatory.iterator().next();
  if (scoringMandatory.contains(required)) {
    subqueries.addAll(decompose(required));
  } else {
    // a FILTER clause matches but does not score: keep its parts non-scoring
    for (Query part : decompose(required)) {
      subqueries.add(new BooleanQuery.Builder().add(part, BooleanClause.Occur.FILTER).build());
    }
  }
}

Trade-offs

• Each part matches the same documents and scores 0, so the parts add back up to IndexSearcher's score in every shape of the repro tests (not for the sibling bugs' shapes). The presearcher extracts the same terms (its query analyzer treats FILTER like MUST), highlighting reports the same terms (the module's highlighting tests pass, and an independent check compared HighlightsMatch hits with and without the fix), and the number of stored parts is unchanged.
• A clause that is both MUST and FILTER (+a #a) still decomposes as scoring, which is its true score.
• Visible side effect: a bare FILTER-only query now scores 0, so ScoringMatch (which keeps only scores above 0) no longer reports it, as it already does not for two-FILTER queries. ExplainingMatch and SIMPLE_MATCHER still report it.
• Each part is wrapped in a one-clause FILTER BooleanQuery, created at registration and rewritten and run with the part at each match; not benchmarked. It does not touch the sibling decomposition bugs (DisjunctionMaxQuery sums, collapsed repeated parts).

What the checks validated

Gradle in a scratch worktree at a9779ef3c193, seed DEADBEEF: the whole :lucene:monitor:test with the 3 repro classes added, 35 classes, 201 tests, 0 failures, 1 skipped (a @Nightly test), with assertions on and off; all 13 repro tests that fail on stock pass. :lucene:monitor:tidy leaves the patch unchanged, and :lucene:monitor:check -x test with the fix and the primary test (google-java-format, ecj lint, forbidden APIs): BUILD SUCCESSFUL. The finder's earlier run of the same change: 180 upstream monitor tests, 0 failures.

A hasMust flag in the clause loop and one early return, if (mandatory.size() == 1 && hasMust == false) return Collections.singleton(q);, plus TestQueryDecomposer.testSingleFilterClauseIsNotDecomposed. +6 lines in QueryDecomposer.java, +13 in the test. The stored query is exactly the registered one.

Trade-offs

A FILTER clause over a disjunction, #(a b ...), becomes one stored entry instead of one per disjunct, so it runs whole whenever selected and Monitor.getDisjunctCount() drops for such queries, and the query index holds and runs fewer entries for them. Matching is unchanged. Not benchmarked.

What the checks validated

The same gradle runs: 35 classes, 201 tests, 0 failures, 1 skipped, with assertions on and off; :lucene:monitor:check -x test: BUILD SUCCESSFUL.

Keep the split for the presearcher, but when any part is selected run the whole registered query. Written for the sibling bug; its whole-module run includes this bug's finder test, which passes.

Trade-offs

Also fixes the two sibling decomposition bugs, but changes every matcher and the meaning of the queries-run count, and keeps a reference to the whole query in every cache entry. A design change for maintainers to weigh.

Say that the Monitor scores the decomposed parts and that a lone FILTER clause is scored.

Trade-offs

A legitimate maintainer choice, but ExplainingMatch would keep contradicting IndexSearcher.explain. Workaround today: a QueryDecomposer subclass that returns such a query whole, set with MonitorConfiguration.setQueryDecomposer.

Method: bounded-exhaustive translation validation against a reference semantics (the hunt's "semi-formal" tier).

• The hunt's planning probe for this area compared ScoringMatch with IndexSearcher and gave the first signal: #apple scored 0 in a search but 0.13076457 in the Monitor.
• The finder's harness ran six matchers (simple, scoring, explaining, highlighting, parallel, partition) on the stock decomposer, and three of them on six decomposer variants, against IndexSearcher on a one-document index: every depth-1 query (420) on 55 documents, every depth-2 query (7,685) on 19 documents, and 3,000 random depth-3 queries on 34 documents; about 6.5 million checks with assertions on and the same with them off.
• Root attribution with public decomposer subclasses: of the harness's named variants, the FILTER variant (it never splits a BooleanQuery with a FILTER clause, which is broader than fix A) was the only one that on its own removed 348, 5,060 and 2,468 score-mismatch rows in the three tiers, for the scoring and the explaining matcher each; further rows were removed by it together with a sibling variant. Other variants explain the other families: the two sibling bugs (DisjunctionMaxQuery sums, collapsed repeated parts) and a separately registered minimumShouldMatch bug.
• A Z3 model of the score algebra over the reals confirmed that the single-FILTER rule preserves the score only when the clause scores 0.
• The finder then wrote the public-API JUnit test, and independent checkers reproduced it and widened it (the nested and FILTER-over-disjunction forms).