Surround query parser ignores a ^ weight written directly on a field prefix's operand, as in title: lucene^3

In the surround query language you weight a clause with ^N and restrict it to a field with a prefix such as title:. When the weight is written on the operand of the field prefix, as in title: lucene^3, the parser stores it on the operand and prints it ((title:lucene^3.0)), but FieldsQuery, the node for the prefix, builds its operand with makeLuceneQueryFieldNoBoost, so no BoostQuery is ever created and the clause scores as if unweighted. Every operand shape that was run loses it: a term, a term under two prefixes, a prefix OR, a distance query and a parenthesised query; other operands reach the same line (from the code, not run). Only scores and ranking change: the same documents match, and the reproduced queries run without an error. Inside a larger query the ranking flips: with A matching in its title and B in its body, or(title: lucene^3, body: lucene) scores A 0.291 (unweighted) below B 0.433, while or((title: lucene)^3, body: lucene) ranks A first (0.874). The boolean composites (OR, AND, NOT) keep their children's weights, so title: lucene^3 or search works while title: lucene^3 does not. The call dates from the parser's first commit (2005), so every release since 1.9 is affected. A one-line fix (makeLuceneQueryField instead of makeLuceneQueryFieldNoBoost) makes the repro pass and keeps the whole lucene/queryparser suite green.

• Surround query parser: an opt-in parser in lucene/queryparser for span-based queries: W and N proximity, AND, OR, NOT, truncation, field prefixes and ^ weights, in infix (a or b) and prefix (or(a, b)) form. You call QueryParser.parse(text), then makeLuceneQueryField(defaultField, new BasicQueryFactory()).
• Field prefix: title: q limits q to the field title; title: body: q means q in either field (an OR over the fields). Internally a FieldsQuery node wraps the operand q.
• Weight: ^N multiplies a clause's score by N. In Lucene a weight becomes a BoostQuery around the clause.

1. Where the weight goes. The grammar parses a field query as optional field names followed by an OR query (QueryParser.jj:251-258), and attaches ^N to a primary query: a term, a prefix-operator query or a parenthesised query (:376-385, OptionalWeights :456-469). In title: lucene^3 the weight sits on the term lucene; in (title: lucene)^3 it sits on the FieldsQuery.
2. How weights are applied. SrndQuery.makeLuceneQueryField builds a node with makeLuceneQueryFieldNoBoost and wraps it in a BoostQuery when the node is weighted ("weight may be at any level in a SrndQuery", SrndQuery.java:58-63). OR, AND and NOT build each child with makeLuceneQueryField (ComposedQuery.java:64-71; "subqueries can be individually boosted", OrQuery.java:34, AndQuery.java:32). The proximity composites (W, N) differ: DistanceQuery also extends ComposedQuery but builds a DistanceRewriteQuery (DistanceQuery.java:29, 100-101) whose span clauses ignore the children's weights (SpanNearClauseFactory.java:123-134), so 2w(lucene^3, tuning) scores like 2w(lucene, tuning): the separate, known span case (LUCENE-8143). A weight around a whole distance query, (body: 2w(lucene, search))^2, is applied.
3. The exception. FieldsQuery.makeLuceneQueryNoBoost builds its operand with q.makeLuceneQueryFieldNoBoost(fieldNames.get(0), qf) (FieldsQuery.java:51-54), which skips the operand's BoostQuery. It is the only place in the surround package that builds a child with the NoBoost variant.
4. Two or more fields. The multi-field branch (:55-67) clones the weighted operand into one unweighted FieldsQuery per field inside an OR; each copy reaches line 54, so the weight is dropped there too.
5. Nothing shows it. The parsed query prints the weight ((title:lucene^3.0)), and so does the Lucene query, whose toString embeds the surround term (SimpleTermRewriteQuery(title, lucene^3.0, ...)); but no BoostQuery is in the tree.

Two documents, the default IndexWriterConfig (StandardAnalyzer) and similarity (BM25), default field body: A has title lucene in action, B has body lucene tuning and lucene scoring.

With searcher.rewrite(q), the bug form becomes title:lucene body:lucene, the same query as without the weight; the control becomes (title:lucene)^3.0 body:lucene. The same numbers with assertions off, on 10.5.2, and, with the fix, the bug form scores like the control.

The repro test uses four documents, StandardAnalyzer, BM25, default field title. Doc 0: lucene in a ten-word title; doc 1: body lucene; doc 2: lucene search in both fields; doc 3: filler. The test does not print the control scores; a main() print of the same index does (the first row's expected column), and it gives the bug form exactly the test's numbers.

In the hunt's two-document index, or(title: lucene^3, body: lucene) gives the title document 0.13076457 instead of 0.39229375, a tie with the body document instead of first place.

• Who. Applications that pass query text to the surround QueryParser (lucene/queryparser; opt-in and niche; Solr exposes it as {!surround}) and weight a field-prefixed clause in the standard-parser style the README points to (it lists field:termtext and ^ boost as separate items and has no weighted example): field: term^N, field: (...)^N, field: or(...)^N, field: 2w(...)^N, with one field or several. A 2017 solr-user post used exactly that shape, field1:(W(X, A) OR ...)^10, in a boost query, and nobody in the thread noticed the weight.
• Effect. Silent: the weighted clause scores as if unweighted. At top level only absolute scores change (every hit is off by the same factor); inside a disjunction or conjunction with other clauses the ranking changes (the flip above). The set of matching documents and the total hit count never change, because weights must be greater than 0; a changed order can change which documents make a top-N cut. The reproduced queries run without an error.
• Default vs opt-in. The diagram's probe uses the default IndexWriterConfig (StandardAnalyzer), codec, searcher and similarity (BM25). The JUnit repro sets StandardAnalyzer and BM25 explicitly and runs under the test framework's randomized codec (CheapBastard in the cited runs); a default-codec print of its index gives the same scores. Assertions on and off. The surround parser itself is opt-in.
• Scale. None needed: an index of two documents shows it, deterministically, every time the syntax is used.
• How likely. field: term^N is the natural way to weight a field clause and fails every time; but the parser is niche, only scores change, and nobody reported it in 21 years.
• Workaround. Put the weight on the parenthesised field query: (title: lucene)^3.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: or(title: lucene^3, body: lucene) expected:<0.39229375> but was:<0.13076457>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:577)
       org.apache.lucene.queryparser.surround.query.TestSrndFieldPrefixOperandWeight.testWeightOnFieldPrefixOperand(TestSrndFieldPrefixOperandWeight.java:46)

• The documented syntax. The parser javadoc lists "boosting (via ^)" (QueryParser.jj:58-62). The README introduces "Query terms from the Lucene standard query parser: field:termtext, ^ boost" (docs/surround/README.txt:22-25) with field-prefix examples such as title: text: aa (:44-47). In the classic parser, field:term^N wraps that clause in a BoostQuery (classic QueryParser.jj:372, QueryParserBase.java:813-829).
• The code's own intent. "weight may be at any level in a SrndQuery" (SrndQuery.java:61); "subqueries can be individually boosted" (OrQuery.java:34, AndQuery.java:32). The parser stores the weight and toString prints it; under a field prefix, no query-building or scoring code applies it (only toString reads it).
• The author's description. In LUCENE-6590 (#7648, 2015) the parser's author wrote that for OR queries "the given boosts are passed to the generated boolean query clauses and they are used there as expected", and that "there are no tests for expected score values of surround queries". The weights they named as ignored were those inside proximity (span) operands, a different case (LUCENE-8143); here the weight is on the whole operand, and (body: 2w(a, b))^2 works while body: 2w(a, b)^2 does not.
• Inconsistency no design would choose. title: lucene^3 or search applies the weight (the term is a child of the OR), title: lucene^3 and title: (lucene or zzz)^3 drop it, and adding a prefix equal to the default field cuts the score of lucene^3 from 0.6084424 to 0.20281413.
• No test pins it. The upstream surround tests contain no ^ at all.
• What is inferred. No document says that f: q^N equals (f: q)^N: the README lists field:termtext and ^ boost separately and has no weighted field example. The expectation rests on the standard-parser syntax the README points to, the parser's own comments and the behaviour of the boolean composites.

The strongest case for "intended" (an independent reviewer from another model family rated it moderate): the grammar attaches the weight to the operand before FieldsQuery wraps it, so title: a^4 and (title: a)^4 are different trees; FieldsQuery calls the NoBoost variant explicitly, and has since the 2005 contribution; the parser's author kept that call in an unmerged 2016-17 rework (LUCENE-7613, #8664, Won't Fix in 2018), also for its per-field DisjunctionMaxQuery clauses, in a patch whose description does discuss boosts ("This uses SpanBoostQuery when proximity (sub)queries are boosted. There is no effect on the scores yet"); FieldsQuery is commented "mostly untested" and the README says "Not tested: multiple fields"; nothing documents f: q^N as equal to (f: q)^N; and a fix changes the scores of queries that have run this way for 20 years. Separately, open #16820 proposes deprecating rarely maintained parsers for Lucene 11: a won't-fix risk, not an intent argument.

Why it fails: no document or test says the weight is ignored, and no query-building code applies it after the parser stores and prints it. The "untested" comments explain why nobody noticed, not a decision. The LUCENE-7613 patch discusses boosts only for proximity subqueries, never the weight on a field prefix's operand; its surround tests have no ^ and no score assertion, and it says it "does not affect the syntax". That it simply carried the old call forward is an inference, not something the patch says. The README lists field:termtext and ^ boost among its terms from the standard parser, where a field-qualified boost works, and the boolean composites (OR, AND, NOT) keep their children's weights. The span limitation the author did name is a different, discussed case. The score change is the fix and belongs in a CHANGES entry. Verdict: not intended, a defect (the independent reproducer, the cross-model reviewer, the devil's advocate and the adjudication all agree).

One fix was applied in a scratch checkout at a9779ef3c193 and run through gradle; the alternatives were not run. The score change is the point of the fix, so a PR wants a CHANGES entry.

Workaround today: write the weight on the parenthesised field query, (title: lucene)^3.

Related, separate (not part of this bug, not verified by the adjudication): FieldsQuery.toString never appends the FieldsQuery's own weight (FieldsQuery.java:84-92), so the surround objects for or((title: a)^3, z) and or((title: a), z) print and compare equal (SrndQuery.equals compares toString). Lucene Query equality is not affected. Executed by the devil's advocate's probe; a weightToString(r) call would fix it (not run).

Call q.makeLuceneQueryField(fieldNames.get(0), qf) in the single-field branch of FieldsQuery.makeLuceneQueryNoBoost (FieldsQuery.java:54). The multi-field branch builds per-field FieldsQuery copies that reach the same line, so it is fixed too. Weights compose by multiplication: in (title: lucene^3)^2 the operand weight and the outer FieldsQuery weight give BoostQuery(BoostQuery(q, 3), 2), as OptionalWeights already multiplies lucene^3^2 (QueryParser.jj:469); no weight is applied twice (executed: with the fix (title: lucene^3)^2 scores 6.0x title: lucene, stock 2.0x). In the multi-field form each per-field clause is boosted instead of their OR: the same score (identical floats in the repro).

     if (fieldNames.size() == 1) {
       /* single field name: no new queries needed */
-      return q.makeLuceneQueryFieldNoBoost(fieldNames.get(0), qf);
+      return q.makeLuceneQueryField(fieldNames.get(0), qf);
     } else {

Trade-offs

Behaviour change by design: queries that use this syntax score differently after the fix, so it wants a CHANGES entry. Small structural overhead, not benchmarked: one extra BoostQuery per weighted operand and field (a two-field prefix gets one per field). The hunt's sketch is the same change with a trailing comment. By shadowing it passed the finder's test (2 of 2), the cross-model reviewer's refute-lens class (4 of 4) and the two failing methods of its intended-lens class (the only ones that run selected), the devil's advocate's probe except its separate toString test (8 of 9), and the 67 upstream surround tests; the independent reproducer and the adjudication ran the reproducer's test with its own, identical one-line change (6 of 6).

What the checks validated

Executed in a scratch checkout at a9779ef3c193 through gradle (removed afterwards). Stock: the repro fails 5 of 6 with assertions on and off and 50 of 50 executions over 10 iterations (seed C0FFEE68), while the 67 upstream surround tests pass. With the fix: the repro passes 6 of 6 with assertions on and off and 60 of 60 over 10 iterations; the whole lucene/queryparser module passes (31 classes, 539 tests, 0 failures; 1 test is @Ignored in the source); ecjLintMain, ecjLintTest, checkGoogleJavaFormat and renderJavadoc pass; tidy changes nothing (re-checked with file hashes and git status before and after); both diffs apply cleanly. Not tested: Solr's surround plugin, benchmarks (no other Lucene module uses the surround package).

In getFieldsQuery, transfer the operand's weight to the new FieldsQuery.

Trade-offs

Needs a way to clear a weight (SrndQuery has none) and a JavaCC regeneration; FieldsQuery.toString never prints its own weight, so the printed query would lose the ^3.0.

Throw a ParseException and point users to (field: q)^N.

Trade-offs

Loud instead of silent, but breaks queries that parse today.

Keep makeLuceneQueryFieldNoBoost for the operand, and wrap the finished single- or multi-field query in one BoostQuery with q.getWeight().

Trade-offs

It should give the same scores, but there is one wrapper around the per-field OR instead of one per field, so the query tree, the rewrite output, equals and query-cache keys differ from the recommended fix.

A javadoc and README note: write (field: q)^N; a weight directly after a field prefix is ignored.

Trade-offs

No score change, but the silent drop and the misleading toString stay.

Method: bounded-exhaustive differential and metamorphic translation validation on the real jar (semi-formal tier), in a hunt unit on surround query semantics.

• An independent reference semantics of the surround language (W, N, AND, OR, NOT, field prefixes, truncations) evaluated 33,320 generated queries on 507 two-field documents: 88,620 checks.
• Hit sets (relation C1): 0 failures. Weight relation C6, "for or(S^3, Y), a document that matches S and not Y must score 3x its score under or(S, Y)": 3,336 failures, every one a weight on the operand of a field prefix (operand a term 104, a W query 1,568, an N query 1,664). With the one-line fix shadowed in: 0 failures.
• Then reduced to a public-API JUnit test on the stock main jar (found 2026-10-27). Checked by an independent reproducer, two executed reviews by an independent reviewer from another model family (OpenAI GPT-5.6), an intended-behaviour devil's advocate and an adjudication.