QueryBuilder throws IllegalArgumentException for a synonym weight above 1 (or 0) on a word that shares its position

Since 8.5.0 (LUCENE-9171) QueryBuilder reads the BoostAttribute of query tokens, which the stock DelimitedBoostTokenFilter (SPI name delimitedBoost) sets from word|weight; placed after SynonymGraphFilter it gives weighted synonyms. For a word alone at its position any finite weight >= 0 works: lion => simba|1.5 gives (body:simba)^1.5. But when the weighted word shares its position with another token, as in the equivalence rule leopard, panthera|1.5, QueryBuilder.newSynonymQuery passes the weight unchanged to SynonymQuery.Builder.addTerm, which accepts only (0, 1] (LUCENE-8652: a per-term weight on the term's within-document frequency). So createBooleanQuery("body", "leopard") and QueryParser.parse("spotted leopard") throw IllegalArgumentException: boost must be a positive float between 0 (exclusive) and 1 (inclusive) instead of returning a query, with a message that names neither the word nor the rule. A weight of 0 fails the same way; weights in (0, 1] work on both paths. A quoted phrase over several words does not throw; it drops the weight (the separate bug LFV-584). None of the javadocs and CHANGES entries checked states a limit for token weights. The failure is loud and fail-closed: no wrong results, no index effect. Affected: 8.5.0 through 10.6.0 and main (7 releases run, the others by tag containment or inference). A small change in newSynonymQuery (+10/-2 lines: divide by the largest weight above 1 and boost the whole SynonymQuery by it) makes the repro pass; it is one policy among several.

• Token weight (boost): a float on a query token, carried by BoostAttribute. DelimitedBoostTokenFilter (SPI name delimitedBoost) splits word|weight into the token word and the weight, with Float.parseFloat and no range check (DelimitedBoostTokenFilter.java:26-33, 53-54). It is the only producer of the attribute on token streams in Lucene's non-test sources.
• Stacked tokens: tokens that share a position: the first has a position increment of 1 or more, each one after it 0. The equivalence rule leopard, panthera|1.5 turns the query word leopard into panthera (weight 1.5, increment 1) and leopard (weight 1, increment 0) at one position; a replacement with one output, lion => simba|1.5, does not stack.
• SynonymQuery: scores several terms as if they were one term. Since LUCENE-8652 (8.1.0) each term can carry a weight that multiplies its within-document frequency (the javadoc says "document frequencies"); addTerm rejects NaN, values <= 0 and values > 1 (SynonymQuery.java:90-94).

1. createFieldQuery marks hasSynonyms when a token has position increment 0. For text that is not a token graph it sends one position with several tokens (quoted or not) to analyzeBoolean, unquoted text over several positions to analyzeMultiBoolean, and quoted text over several positions to analyzeMultiPhrase when a token is stacked (QueryBuilder.java:308-365).
2. A position with one token becomes newTermQuery(term, boost), a BoostQuery for any weight other than 1 (:638-644); BoostQuery accepts any finite value >= 0 (BoostQuery.java:38-44).
3. A position with several tokens becomes newSynonymQuery, which calls builder.addTerm(t.term, t.boost) with the raw weight (:603-609, the call at :606). It is reached from analyzeBoolean (:385-396), add for analyzeMultiBoolean (:408-409) and analyzeGraphBoolean (:554-555).
4. SynonymQuery.Builder.addTerm throws for 1.5 or 0 (SynonymQuery.java:90-94). Nothing catches it: QueryParserBase.parse converts only ParseException, TokenMgrError and TooManyClauses (QueryParserBase.java:129-146).

So one attribute means two things: a score multiplier (any finite value >= 0) for a word alone at its position, and a frequency weight in (0, 1] for words that share one. Values legal on the first path throw on the second.

Index of four one-word documents (leopard, panthera, simba, cat), BM25, main 645d4fe14a20: leopard, panthera|0.5 gives Synonym(body:leopard body:panthera^0.5) (doc leopard 0.5473, doc panthera 0.3541); |0.9 and |1 work too; lion => simba|1.5 gives (body:simba)^1.5 (0.8209); lion => simba|0 gives (body:simba)^0.0 (match, score 0.0). leopard, panthera|1.5, leopard, panthera|0 and lion => lion, simba|1.5 throw, for leopard, spotted leopard and QueryParser.parse("spotted leopard"). A quoted "spotted leopard" does not throw: it becomes body:"spotted (panthera leopard)" and loses the weight (the separate bug LFV-584).

• Who is hit. Applications whose query-time analyzer emits token weights: weighted synonyms (synonymGraph then delimitedBoost, the chain the Solr Reference Guide documents) or user-typed word|weight (delimitedBoost before the synonym filter, where tiger|2 works and leopard|2 throws). The trigger is a weight above 1, or exactly 0, on a word the analyzer stacks with another token: an equivalence rule such as leopard, panthera|1.5, or a replacement with several outputs such as lion => lion, simba|1.5.
• Entry points (all executed): createBooleanQuery, createMinShouldMatchQuery, a single-position createPhraseQuery, the classic QueryParser and SimpleQueryParser.
• Effect. Building a query from unquoted text that contains the word fails, through each entry point above (the single-position createPhraseQuery is the one-word phrase); a quoted phrase over several words does not throw but drops the weight (LFV-584). Fail-closed: no query object is returned, so no restriction or exclusion clause can be lost (4 of 4 property tests); no index effect. The message names neither the word nor the rule.
• Default vs opt-in. The weight-emitting query analyzer is opt-in; writer, codec, similarity, searcher and parsers are defaults. No scale is needed: one word and one rule.
• How likely. Rare: every weighted-synonym example found in the recorded searches uses weights of at most 1, the failure shows at the first unquoted query that contains the word, and the recorded searches found no report of it since 8.5.0 (2020).

The test asserts only the claim: building the query throws instead of returning one. It does not assert the query's shape, because upstream could fix this in more than one way (normalize the weights, reject them early with a clear message, or document a limit). The test uses no randomness of its own, and QueryBuilder reads none of the settings LuceneTestCase randomizes; it fails the same with seeds DEADBEEF, 5EED5881 and 5EED5882. It needs the analysis-common module, so it lives in org.apache.lucene.analysis.boost.

As the regression test: the same test (for example next to TestDelimitedBoostTokenFilter). It fails on stock and passes with the recommended fix; with the reject-early option it would become an expectThrows on the new message.

cp TestStackedTokenBoostRange.java $LUCENE/lucene/analysis/common/src/test/org/apache/lucene/analysis/boost/
cd $LUCENE && ./gradlew :lucene:analysis:common:test --tests org.apache.lucene.analysis.boost.TestStackedTokenBoostRange -Ptests.seed=DEADBEEF
# assertions off: add -Ptests.asserts=false (same result)

TestStackedTokenBoostRange > testSynonymWeightAboveOne FAILED
    java.lang.AssertionError: expected a query for synonym weight 1.5, but got java.lang.IllegalArgumentException: boost must be a positive float between 0 (exclusive) and 1 (inclusive)
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:6BCD06C43690A2DD]:0)
        at org.apache.lucene.analysis.boost.TestStackedTokenBoostRange.testSynonymWeightAboveOne(TestStackedTokenBoostRange.java:51)

        Caused by:
        java.lang.IllegalArgumentException: boost must be a positive float between 0 (exclusive) and 1 (inclusive)
            at org.apache.lucene.search.SynonymQuery$Builder.addTerm(SynonymQuery.java:92)
            at org.apache.lucene.util.QueryBuilder.newSynonymQuery(QueryBuilder.java:606)
            at org.apache.lucene.util.QueryBuilder.analyzeBoolean(QueryBuilder.java:395)
            at org.apache.lucene.util.QueryBuilder.createFieldQuery(QueryBuilder.java:360)
            at org.apache.lucene.util.QueryBuilder.createFieldQuery(QueryBuilder.java:256)
            at org.apache.lucene.util.QueryBuilder.createBooleanQuery(QueryBuilder.java:108)
            at org.apache.lucene.util.QueryBuilder.createBooleanQuery(QueryBuilder.java:93)

# gradle, main 645d4fe14a20, seed DEADBEEF; the same with -Ptests.asserts=false
# (module prefixes org.apache.lucene.core@11.0.0-SNAPSHOT/ trimmed)

• The feature's contract. CHANGES 8.5.0, LUCENE-9171: "QueryBuilder can now use BoostAttributes on input token streams to selectively boost particular terms or synonyms in parsed queries" (CHANGES.txt:5965-5966), and "QueryBuilder.newTermQuery() and .newSynonymQuery() now take boost parameters" (:5925-5926). No range is stated.
• The producer states no range. DelimitedBoostTokenFilter: "Characters before the delimiter are the "token", those after are the boost", parsed with Float.parseFloat; its factory is SPI-registered as delimitedBoost.
• Stacking decides. lion => simba|1.5 gives (body:simba)^1.5; leopard, panthera|1.5 and lion => lion, simba|1.5 throw. A user cannot tell from the configuration which rules are safe.
• SynonymQuery documents no range either. The addTerm javadoc says only that "document frequencies of this term will be boosted by boost" (SynonymQuery.java:86-89); the scorer actually scales the within-document frequency (:649-685).
• No test. TestQueryBuilder#testTokenStreamBoosts (TestQueryBuilder.java:572-605) uses 0.5 only; no test asserts the exception at the QueryBuilder level.
• History. The (0, 1] check (LUCENE-8652, 2019; in 8.1.0 through the branch_8x backport 24427f9f5cd) predates token boosts and was harmless while QueryBuilder passed none; LUCENE-9171 (2020; in 8.5.0 through the backport 2752d50dd1d) passed them through without range handling. The reviews of LUCENE-9171, SOLR-12238 and apache/lucene-solr#357 never discuss a range.

The strongest case for "intended" (the refute-lens review, the works-as-designed review and the intended-behaviour devil's advocate): SynonymQuery's range is deliberate. LUCENE-8652 added it as a "deboost" (CHANGES.txt:6549-6550); its author (jimczi) wrote "The boost must be between 0+ and 1 in order to ensure that document term frequencies do not go beyond the total term frequency" (#9698); TestSynonymQuery#testBogusParams expects 1.3f to throw. BoostAttribute is @lucene.internal, and its javadoc says it "is intended to be added only by the TermsEnum to itself in its constructor and consumed by the MultiTermQuery.RewriteMethod" (BoostAttribute.java:24-35); in #12169 a committer (rmuir) objected to any use of it in token streams. Every example in the recorded searches uses weights of at most 1, and newSynonymQuery is an overridable extension point (an override accepts 1.5).

Why it fails: the range is SynonymQuery's own invariant and stays valid; the defect is one layer up. LUCENE-9171 made token weights documented QueryBuilder input, shipped a public, SPI-registered producer that parses any float, and applies the same attribute without a limit when the word is alone at its position. The BoostAttribute javadoc predates that feature, and the #12169 objection argues against the feature as a whole, not for a limit that depends on stacking. No API promises either that a weight above 1 on a stacked token must be accepted, so the claim is narrower: an undocumented limit that depends on stacking, which surfaces as an unchecked exception. Even if upstream rules that synonym weights must be in (0, 1], the current failure is the wrong error: an internal exception that names neither the word nor the rule. Hence the class "wrong error", not "crash". Verdict: not intended; the open question is which layer should handle the range.

Upstream should choose the policy: normalize, reject early with a clear message, or fall back to boosted term clauses. The first one was run through gradle in a scratch worktree of 645d4fe14a20.

+10/-2 in QueryBuilder.java. Divide every weight by the largest one above 1 and wrap the SynonymQuery in a BoostQuery of that value; a weight of 0 becomes Float.MIN_VALUE, so the term still matches, as (term)^0 does.

Result: leopard, panthera|1.5 gives (Synonym(body:leopard^0.6666667 body:panthera))^1.5; doc panthera scores 0.8209 (the score of (body:simba)^1.5), doc leopard 0.6450. QueryParser.parse("spotted leopard") gives body:spotted (Synonym(body:leopard^0.6666667 body:panthera))^1.5. leopard, panthera|0 gives Synonym(body:leopard body:panthera^1.4E-45): doc panthera matches with score 0.0. Negative, NaN and infinite weights still throw today's exception.

float max = 1f;
for (TermAndBoost t : terms) {
  max = Math.max(max, t.boost);
}
SynonymQuery.Builder builder = new SynonymQuery.Builder(field);
for (TermAndBoost t : terms) {
  builder.addTerm(t.term, t.boost == 0f ? Float.MIN_VALUE : t.boost / max);
}
Query query = builder.build();
return max == 1f ? query : new BoostQuery(query, max);

Trade-offs

• Not score-equivalent to "panthera counts 1.5 times": SynonymQuery blends frequencies, which BM25 saturates, so the outer 1.5 also raises a document that contains only leopard (0.5473 with weight 1, 0.6450 with the fix). It is one policy; dividing without the outer boost, or clamping to 1, are others.
• Gives weight 0 a meaning (match, score about 0) where it throws today.
• Changes the query (shape, toString, equals) only for weights that throw today, so no working configuration changes. It also needs a CHANGES entry; no benchmark was run.
• The hunt's first sketch did the same scaling but also turned negative and NaN weights into Float.MIN_VALUE silently; this version keeps them throwing.

What the checks validated

• Gradle in a scratch worktree of 645d4fe14a20 (removed afterwards), seed DEADBEEF: without the fix the repro fails with assertions on and off; with the fix it passes with assertions on and off (1 of 1 each).
• Closest suites with the fix: TestQueryBuilder, TestSynonymQuery, the analysis-common boost and synonym packages (the repro included) and the whole lucene/queryparser module: 45 classes, 692 tests, 0 failures, 0 errors, 1 skipped.
• :lucene:core:check -x test (google-java-format, ecj lint, forbidden APIs) and the analysis-common format check pass with the fix and the repro in place.
• A probe on the main jars with the fixed QueryBuilder shadowed gives the queries and scores listed above; weights in (0, 1] give exactly today's queries and scores. Other modules' suites, Solr and benchmarks were not run.

In newSynonymQuery, check each weight and throw an IllegalArgumentException that names the term, its weight and the (0, 1] limit for words that share a position; state the limit in the DelimitedBoostTokenFilter and QueryBuilder javadocs.

Trade-offs

No scoring policy to choose and no score change, but the query still fails (now with a message an operator can act on), and it still escapes QueryParser.parse unchecked unless the parser converts it. The repro would become an expectThrows on the message. Not run.

When a weight at a position is outside (0, 1], build a BooleanQuery of SHOULD (term)^weight clauses for that position, as Solr's as_distinct_terms synonym style does (read in Solr's source, not run).

Trade-offs

Keeps each weight exactly as a score multiplier, like the unstacked path, but loses SynonymQuery's blended statistics for that position, so the query's shape and scoring change with the weight's value. Not run.

State in the javadocs that weights on stacked tokens must be in (0, 1].

Trade-offs

Leaves the internal exception, whose message names neither the word nor the rule. Still a credible upstream outcome: BoostAttribute is @lucene.internal, and SynonymQuery's bound is deliberate and tested.

Method: bounded-exhaustive translation validation on the real jar (semi-formal tier), in a hunt unit on QueryBuilder reference semantics (2026-10-26, main a9779ef3c193).

• Scope C: every well-formed token stream of at most 3 tokens over {a, b, c} (19,458 of 106,956 enumerated; the rest are ill-formed graphs) with position increments 0-2, position lengths 1-2 and weights {0.5, 1, 2}, through 10 QueryBuilder entry points, compared with an independent reference model: 739,080 checks, 155,052 of them threw this IllegalArgumentException.
• Scope C0: over {a, b} with weights {0, 0.25, 1, 1.5}: 125,600 checks, 42,176 threw it.
• Every throwing check came from a stream in which a stacked token carries a weight outside (0, 1]; no other exception class occurred.
• Then public-API JUnit tests on stock analysis chains (SynonymGraphFilter and DelimitedBoostTokenFilter in either order) and through QueryParser and SimpleQueryParser; later cut to the one-test repro above.