QueryBuilder drops every token boost from a quoted phrase when one of its words has a single-word synonym

Since 8.5.0 (LUCENE-9171) QueryBuilder reads the BoostAttribute of query tokens, which the stock DelimitedBoostTokenFilter sets from word|weight input, so typed boosts and weighted synonyms such as cat, kitty|0.5 reach the parsed query. For a phrase it multiplies the token boosts: "fast|0.5 tiger" becomes (body:"fast tiger")^0.5, and a phrase with a multi-word synonym becomes one boosted query per path. But when a quoted phrase analyzes to more than one position with a stacked token (a word with a single-word synonym) and no token graph (no multi-word synonym), createFieldQuery builds a MultiPhraseQuery in analyzeMultiPhrase, which never reads BoostAttribute, and every boost of the phrase is silently dropped: "fast|0.5 car" with car, automobile becomes the unboosted body:"fast (automobile car)" and scores 0.2615 instead of 0.1308, and with cat, kitty|0.5 the quoted "cat food" scores kitty food the same as cat food (0.7130 both), although the unquoted query ranks it lower. The classic QueryParser shows the same. The bug changes no match set (a boost only scales scores); only scores and ranking are wrong. Every release from 8.5.0 to 10.6.0 (10.6.0 by reading) and main; the boost-emitting query analyzer is an opt-in choice. A 28-line fix (boost the MultiPhraseQuery like analyzePhrase does, and let the classic parser's explicit-slop rewrite look inside the BoostQuery) fixes the lost boost; the closest suites pass, and only the weighted-synonym test still fails, which this fix leaves to an expansion or a javadoc note by design.

• Token boost: a weight attached to one query token through BoostAttribute. The stock DelimitedBoostTokenFilter (SPI name delimitedBoost) sets it from word|weight and strips the suffix: fast|0.5 becomes the token fast with boost 0.5. With a synonym file such as cat, kitty|0.5 placed before it in the chain, it gives weighted synonyms.
• Single-word synonym: car, automobile adds automobile at the same position as car (position increment 0, a stacked token). A multi-word synonym such as guinea pig, cavy makes a token graph instead.
• QueryBuilder: turns analyzed query text into a Query. The classic QueryParser, MultiFieldQueryParser and SimpleQueryParser build their queries through it. For a quoted phrase it picks one of three builders: analyzePhrase (no stacked token), analyzeGraphPhrase (a token graph) or analyzeMultiPhrase (a stacked token and no graph).

1. createFieldQuery marks hasSynonyms when a token has position increment 0 and sends a quoted, non-graph phrase with such a token to analyzeMultiPhrase, any other quoted, non-graph phrase to analyzePhrase (QueryBuilder.java:347-355).
2. analyzePhrase multiplies the token boosts, phraseBoost *= boostAtt.getBoost(), and returns new BoostQuery(query, phraseBoost) when the product is not 1 (:436-461). analyzeGraphPhrase sends each path of the graph back through createFieldQuery, so a path of several tokens is built by analyzePhrase and keeps its product, and a one-token path is built by analyzeTerm, which reads the boost too (:565-583).
3. analyzeMultiPhrase collects the terms of each position into a MultiPhraseQuery and returns it as is; it never adds or reads BoostAttribute (:463-497). Every other analyze* method reads it, directly or (for analyzeGraphPhrase) through the builders it calls.
4. So whether a boost survives depends on the builder: in practice, on whether some word of the phrase has a single-word synonym (and none a multi-word one). Two cases: (a) a boost on a word without a synonym is lost because another word has one, the clear-cut case; (b) weighted single-word synonyms are ignored inside quotes, weaker because MultiPhraseQuery has no per-term weights.
5. A boost only scales scores, so the matched documents are the same: on 14,376 multi-phrase-path queries (main with and without assertions, 10.5.2) the match sets never differed and 6,461 scores did (executed).

Stock WhitespaceTokenizer, DelimitedBoostTokenFilter and SynonymGraphFilter, BM25, small indexes:

• No synonym: "fast tiger" gives body:"fast tiger", score 0.2615; "fast|0.5 tiger" gives (body:"fast tiger")^0.5, score 0.1308.
• Multi-word synonym guinea pig, cavy: "fast|0.5 guinea pig" gives (body:"fast cavy")^0.5 (body:"fast guinea pig")^0.5, half the score of the unboosted phrase.
• Single-word synonym car, automobile: "fast|0.5 car" gives body:"fast (automobile car)", score 0.2615 (expected 0.1308).
• Weighted synonym cat, kitty|0.5, unquoted cat food: Synonym(body:cat body:kitty^0.5) body:food, cat food 0.3979 and kitty food 0.2867.
• Quoted "cat food": body:"(kitty cat) food", both documents 0.7130.
• Classic QueryParser with tv => television|0.9: "tv food" gives (body:"television food")^0.9, but "tv car" gives body:"television (automobile car)", without the 0.9.

• Who. Users of QueryBuilder.createPhraseQuery, of quoted phrases in the classic QueryParser (executed) and in MultiFieldQueryParser and SimpleQueryParser (executed by the fact check), or of autoGeneratePhraseQueries, whose query-time analyzer sets token boosts. The only stock token-stream producer of BoostAttribute is DelimitedBoostTokenFilter: weighted synonyms from the synonym file, or user-typed word|weight when the tokenizer keeps the delimiter.
• When. Every quoted (or auto-generated) phrase that analyzes to more than one position with a stacked token and no token graph (in practice: some word has a single-word synonym and none a multi-word one), deterministically. A single-word synonym such as car, automobile is an ordinary synonym-file entry.
• Effect. Silent wrong scores and ranking: the phrase's typed boosts and synonym weights are ignored. The matched documents are unchanged, so filters, deny rules and counts built from such a query are not affected; an application that drops hits below a score threshold can still show a different result list. No error.
• Default vs opt-in. Default writer, codec, similarity, searcher and parsers; the boost-emitting query analyzer is an opt-in choice.
• Scale. None needed: one document and one phrase show it.
• In practice a low-severity ranking defect; the searches found no report since 8.5.0 (2020).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: body:"fast (automobile car)" expected:<0.5> but was:<1.0>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:577)
       org.apache.lucene.util.TestQueryBuilderMultiPhraseBoost.testPhraseBoostKeptWhenAWordHasASynonym(TestQueryBuilderMultiPhraseBoost.java:47)

• The contract. CHANGES 8.5.0, LUCENE-9171: "QueryBuilder can now use BoostAttributes on input token streams to selectively boost particular terms or synonyms in parsed queries" (CHANGES.txt:5948-5949).
• QueryBuilder's own rule. analyzePhrase multiplies the token boosts and graph phrases keep them per path; the same boosted token is honoured or dropped depending only on whether an unrelated word has a single-word synonym ("tv food" keeps ^0.9, "tv car" drops it, executed through the classic parser).
• The feature author meant phrases. In the review of apache/lucene-solr#357, a reviewer (romseygeek) wrote "we need to combine boosts together somehow; currently your phrase boost is just the boost of the last term in the phrase", and the author (alessandrobenedetti) answered "I implemented a simple multiplicative boost ... it's also compatible in not synonym cases, if the user needs a boost per token in phrase and span queries". MultiPhraseQuery is never mentioned in that PR, in LUCENE-9171 (#10211) or in SOLR-12238.
• History: an omission. Commit 663611c99c7 added BoostAttribute handling to analyzeTerm, analyzeBoolean, analyzeMultiBoolean, analyzePhrase, analyzeGraphBoolean and createSpanQuery, but not to the older analyzeMultiPhrase (2013/2015), which has had no boost-related change since.
• Tests. The only upstream test of token-stream boosts through QueryBuilder, TestQueryBuilder#testTokenStreamBoosts, covers createBooleanQuery only (TestDelimitedBoostTokenFilter tests the filter's own output); no test asserts the unboosted MultiPhraseQuery.
• Legal trigger. Public API, the stock DelimitedBoostTokenFilter and SynonymGraphFilter in the documented order, the classic parser.

The strongest case for "intended" (the works-as-designed reviewer, the intended-behaviour checker, the adjudication, and a committer's own words): BoostAttribute is @lucene.internal and its javadoc says it is "intended to be added only by the TermsEnum to itself in its constructor and consumed by the MultiTermQuery.RewriteMethod". A committer, rmuir, restated this in 2023 in the review of #12169 (the Word2VecSynonymFilter): "BoostAttribute is not for use in tokenstreams. it is for multitermquery." (review comment 1116841545, 2023-02-24); "This is wrong, wrong wrong. Remove any use of boostattribute outside of MultiTermQuery. It documents not to do this." (review comment 1121546026, 2023-03-01); "It is really important that this only be used by fuzzyquery, it is only for multitermquery. That's why it is documented as such." (issue comment 1467945170, 2023-03-14). The filter merged without setting it, and its author's follow-up, open #12433, proposes a separate token-stream boost attribute next to a renamed MultiTermQuery one. Further, createPhraseQuery's javadoc lists the query types it returns and promises nothing about boosts; the CHANGES entry only says "can now use"; LUCENE-9171 was scoped to term and synonym queries (a reviewer suggested to "leave the graph queries alone for the moment, as boosting there is more complex"); and MultiPhraseQuery cannot weight alternatives. So a maintainer may well answer that token streams should not set this attribute at all. A 2011 proposal, LUCENE-3130 (#4203), already said the multi-term synonym phrase case would need a decision ("possibly just punt for now").

Why it fails: the javadoc conflicts with the shipped feature. LUCENE-9171 made token-stream boosts documented QueryBuilder input (the CHANGES 8.5.0 entry), with a public SPI filter and a reference-guide recipe, and QueryBuilder still reads the attribute on every other path at a9779ef3c193. rmuir's statements argue against the feature as a whole, not for dropping boosts on one path only: while the feature ships, a boost kept on one path and dropped on another is a bug; if the maintainers would rather retire or move token-stream boosts (the direction of #12433), the report can point to that decision. The author stated the multiplicative rule is for "a boost per token in phrase and span queries", and the final commit applied it to plain and graph phrases despite the "leave graph alone" remark. No doc, test or discussion records the multi-phrase omission as a choice. For case (a) a BoostQuery around the MultiPhraseQuery expresses the rule exactly, so "MultiPhraseQuery cannot weight alternatives" does not apply. LUCENE-3130 predates the code by nine years and describes no implemented behaviour. For (b) the case for "intended" is stronger (the cross-model fact check rates it moderate): MultiPhraseQuery cannot weight alternatives, and taking the largest, averaging, expanding or rejecting differing weights are all different policies. Verdict: not intended. High confidence for (a); (b) may be treated upstream as an enhancement or a documentation note, so file (a) as the bug and offer (b) as fix-or-document.

One fix was applied in a scratch worktree at a9779ef3c193 and run through gradle; the hunt's expansion sketch was run by shadowing and through gradle; the other options were not run. Lead the PR with case (a); offer (b) as expansion or documentation.

Part A, QueryBuilder.analyzeMultiPhrase: read BoostAttribute; each position contributes the largest boost of its stacked terms; the phrase boost is the product over positions; wrap the MultiPhraseQuery in a BoostQuery when it is not 1. When the terms at each position share one boost (typed boosts on words without synonyms, unweighted synonyms), this is exactly analyzePhrase's rule. Part B, QueryParserBase.getFieldQuery(String, String, int): the classic parser re-applies an explicit ~N only to a bare PhraseQuery or MultiPhraseQuery (QueryParserBase.java:420-432), so once part A wraps the phrase, "fast|0.5 car"~3 would lose the ~3 that stock keeps; part B adds a BoostQuery branch that applies the slop to the wrapped phrase, as MultiFieldQueryParser.applySlop has done since 9.6.0 (#12196). Part B also fixes an older, separate gap: a boosted plain phrase such as "fast|0.5 tiger"~3 loses its ~3 on stock (tracked separately in this hunt's list as the classic parser's explicit-slop bug). A maintainer may want part B as its own commit or issue. If that bug's own fix (pass the explicit slop into the analysis) lands first, part B is not needed. A PR also needs a CHANGES entry.

BoostAttribute boostAtt = stream.addAttribute(BoostAttribute.class);
float phraseBoost = DEFAULT_BOOST;
float positionBoost = Float.NEGATIVE_INFINITY;
// ... at each new position, after mpqb.add(...):
  phraseBoost *= positionBoost;
  positionBoost = Float.NEGATIVE_INFINITY;
// ... for every token:
  positionBoost = Math.max(positionBoost, boostAtt.getBoost());
// after the loop:
phraseBoost *= positionBoost;
Query query = mpqb.build();
return phraseBoost == DEFAULT_BOOST ? query : new BoostQuery(query, phraseBoost);

Trade-offs

• Match-neutral (a BoostQuery only scales scores) and cheap: one float per position, no extra clauses. No benchmark was run. It still changes the query object: its shape, toString, equals (so query-cache identity) and explanations; and a zero product gives a ^0.0 query, which BoostQuery.rewrite turns into a constant-score one (BoostQuery.java:92-94).
• Leaves (b): when the terms at one position carry different boosts, the largest counts, so "cat food" with kitty|0.5 still scores both documents 0.7130 (executed). The same holds for a typed boost on the word that has the synonym, when the boost filter runs before the synonym filter: the injected synonym gets the default boost 1, so "fast car|0.5" stays unboosted (executed). The chosen boost is at least the boost of every combination of alternatives; taking the largest is one policy among several (averaging, the exact expansion below, rejecting differing weights), and a maintainer may prefer another.
• Behaviour change by design: boosted phrases with a single-word synonym score differently after the upgrade; part B also makes "fast|0.5 tiger"~3 keep its ~3.
• Zero, negative and non-finite token boosts on a word without a synonym now behave as on the plain-phrase path (executed): fast|0 gives ^0.0, fast|-1 and fast|NaN throw IllegalArgumentException from BoostQuery, where stock silently drops them. On the word with the synonym (boost filter first) the largest boost of the position counts, so the synonym's default 1 hides a zero or negative boost: "fast car|0" and "fast car|-1" stay unboosted, where "fast tiger|0" gives ^0.0 and "fast tiger|-1" throws; "fast car|NaN" and "fast car|Infinity" still throw (executed).

What the checks validated

• Stock main a9779ef3c193: core 1 of 3 fail (both controls pass), analysis-common 2 of 2, queryparser 3 of 3, with assertions on and off.
• With the fix, assertions on and off: core 3 of 3 and queryparser 3 of 3 pass; analysis-common: case (a) passes, case (b) still fails, as designed.
• Closest suites with the fix (seed DEADBEEF): TestQueryBuilder 26 tests and the repro's 3, 0 failures; the analysis-common boost and synonym packages, 13 classes, 124 tests, 1 failure (case (b)); the whole lucene/queryparser module, 31 classes, 536 tests, 0 failures (1 test skipped by @Ignore upstream); the UnifiedHighlighter tests, 7 classes, 439 tests, 0 failures.
• tidy changes nothing; ecjLint, checkGoogleJavaFormat and renderJavadoc of lucene/core and lucene/queryparser pass; the diff applies cleanly to a9779ef3c193.
• Probe on the main jars with the fix shadowed in: "fast|0.5 car" gives (body:"fast (automobile car)")^0.5 and 0.1308; the classic parser gives (body:"fast (automobile car)"~3)^0.5 for "fast|0.5 car"~3.
• Not run: other modules' suites, Solr, benchmarks.

Uniform positions as above; when the terms at a position carry different boosts, build a BooleanQuery of one boosted PhraseQuery per combination of alternatives, as analyzeGraphPhrase does per path. This also fixes (b): "cat food" with kitty|0.5 becomes (body:"kitty food")^0.5 body:"cat food" and scores cat food 0.3979 and kitty food 0.1990, the order of the unquoted query (executed).

Trade-offs

• The query grows with the product of the alternatives: 10 weighted words give 1,024 clauses, and 11 or more throw TooManyClauses at the default limit, where the phrase parses today (executed).
• With slop above 0 it matched a strict superset of today's documents in 14 of 4,809 probe cases; the three printed ones are sloppy and each repeats a term across positions (executed).
• The classic parser's explicit slop is lost for such phrases even with part B, because the result is a BooleanQuery: "cat food"~3 gives (body:"kitty food")^0.5 body:"cat food", where stock keeps ~3 (executed). The same gap exists today for graph phrases.

What the checks validated

Shadowed onto the main jars: the hunt's repro passes and upstream TestQueryBuilder passes 26 of 26. Through gradle with part B, assertions on: the doc's three tests pass (8 of 8, case (b) included), TestQueryBuilder 26 of 26, the analysis-common boost and synonym packages 124 of 124, and the whole lucene/queryparser module 536 tests with 0 failures.

Add to the createPhraseQuery (or analyzeMultiPhrase) javadoc that the terms stacked at one position of a phrase share one weight, so weighted single-word synonyms are not weighted inside quotes.

Trade-offs

Documentation only; pairs with the recommended fix and leaves (b) as a known limitation.

Set the parser's phrase slop to the explicit ~N around the analysis in getFieldQuery(String, String, int) and restore it afterwards, so QueryBuilder builds every phrase shape with the right slop. This is the fix proposed for the classic parser's explicit-slop bug tracked separately in this hunt's list.

Trade-offs

Also covers graph phrases and the expansion option; it changes two pinned TestQueryParser assertions, according to that bug's validation.

Method: bounded-exhaustive translation validation on the real jar (semi-formal tier), in a hunt unit on QueryBuilder reference semantics.

• Scope C: every well-formed token stream of at most 3 tokens over {a, b, c} (19,458 of 106,956 enumerated; the other 87,498 are ill-formed graphs and were skipped) with position increments 0-2, position lengths 1-2 and BoostAttribute values {0.5, 1, 2}, run through every QueryBuilder entry point (Boolean, minimum-should-match and phrase queries at several slops) on the stock main jar and compared, structurally and by the matched documents, with an independent reference model: 739,080 checks in about 10 s.
• A structural boost check flagged every createPhraseQuery result that is a MultiPhraseQuery while some token carries a boost other than 1: 35,244 checks at slop 0 (26,820 + 8,424), class MULTIPHRASE-BOOST-DROPPED. The check is structural, so it also counts streams whose boosts multiply to 1, where nothing is lost. Plain phrases, and graph phrases without a hole inside a multi-word synonym, kept their boosts. The run's other classes are not this bug: boosts or matches lost around a hole inside a multi-word synonym (the known upstream #9185 family) and an IllegalArgumentException from SynonymQuery, registered separately as LFV-588.
• Then public-API JUnit tests on the stock jar with SynonymGraphFilter and DelimitedBoostTokenFilter confirmed it by scores. Found 2026-10-26 on main a9779ef3c193; then checked by an independent reproducer, an intended-behaviour devil's advocate, reading-only reviews by a model from another family (OpenAI GPT-5.6) and an adjudication.