Flexible query parser prints small boosts in E-notation: ab^0.0009 comes back as ab^9.0E-4, which it cannot parse

The flexible query parser reads a boost ^N into a BoostQueryNode holding a float. To print it, toQueryString() and toString() write a whole-number boost as a long and every other boost with Float.toString, which uses scientific notation below 0.001. So ab^0.0009 prints f:ab^9.0E-4. The grammar's number token has no exponent, so the same parser rejects that text with ParseException; before 9.0.0 it silently read it as a different query (f:ab^9 f:E-4). Above 2^63 the whole-number check overflows and 1e19 prints 1.0E19, with the same result. An exhaustive check ran the real printer on all 2,139,095,040 non-negative finite floats and classified its output with a model of the grammar's number token (validated against the real parser in 761,082 checks): exactly the boosts in (0, 0.001) and above 2^63 print text that does not parse back; every other boost round-trips. That breaks the documented promise that a node tree prints "a query string that can be parsed back by the original text parser". Original parsing and search are unaffected: the Query built from the original text is correct, and the impact appears only when printed node text is reused as query syntax. FuzzyQueryNode has the same pattern in a different method (ab~0.0001 prints ab~1.0E-4), and there the re-parse silently builds a different query. The fix prints a plain decimal (BigDecimal.toPlainString) in both nodes.

• Parse side. The grammar's Boost rule reads '^' <NUMBER> and builds new BoostQueryNode(node, Float.parseFloat(...)) (StandardSyntaxParser.jj:695-702, :115-116). The node stores a float (BoostQueryNode.java:33).
• Print side (BoostQueryNode.java:80-84):

private CharSequence getValueString() {
  Float f = Float.valueOf(this.value);
  if (f == f.longValue()) return "" + f.longValue();   // whole numbers up to 2^63
  else return "" + f;                                  // Float.toString: 9.0E-4, 1.0E19
}

toQueryString returns the child's text + "^" + getValueString() (:91-95), and toString prints <boost value='...'> with the same helper (:86-89).

• The grammar. NUMBER is (<_NUM_CHAR>)+ ( "." (<_NUM_CHAR>)+ )? with no exponent (StandardSyntaxParser.jj:160), while TERM (:161) accepts digits, ., E and -. By longest match 9.0E-4 lexes as one term, so Boost ::= '^' <NUMBER> (:688-703) fails.

Float.toString is plain only for magnitudes in [10^-3, 10^7). Every non-integral float is below 2^23, and integral floats up to 2^63 take the long branch, so [0.001, 2^63] always prints plainly. Below 0.001 the else branch prints 9.0E-4. Above 2^63 longValue() saturates at Long.MAX_VALUE, the equality fails and 1e19 prints 1.0E19. An exhaustive run printed all 2,139,095,040 non-negative finite floats with the real BoostQueryNode and classified each output with a model of the NUMBER token (the model agreed with the real parser in all 761,082 real-parser checks on a sample around the notation switches and at random). It found 981,668,462 failures in (0, 0.001) and 545,259,519 above 2^63, and none elsewhere. A grid through 15 host constructs (term, phrase, group, ranges, wildcard, regexp, ...) gave 5,700 ParseExceptions in 19,620 checks and no silent difference.

FuzzyQueryNode.toQueryString appends "~" + this.similarity (FuzzyQueryNode.java:50-57). The grammar's number after ~ is optional (StandardSyntaxParser.jj:712-729), so f:ab~1.0E-4 re-parses without error as ~ with the default 2 edits plus a separate term 1.0E-4. ab~0.0001 builds f:ab~1; its printed text builds f:ab~2 f:1.0e-4, which matched 1 document instead of 0 on a one-document index. That witness is narrow: the edit count changes only for 1-2 character terms (FuzzyQuery.floatToEdits already gives 2, the cap, for 3 or more, so abcd~0.0001 builds f:abcd~2). The extra clause is there for every term: under the default OR operator it widens the query (f:abcd~2 f:1.0E-4), under AND it narrows it (+f:abcd~2 +f:1.0E-4), and what it matches depends on the analyzer.

BoostQueryNodeBuilder copies getValue() into the BoostQuery (BoostQueryNodeBuilder.java:48), so StandardQueryParser.parse("ab^0.0009", "f") searches with boost 0.0009. Apart from the node printers themselves (composite nodes print their children; IntervalQueryNode.toString() returns its own toQueryString), Lucene puts toQueryString output only into exception messages (for example QueryTreeBuilder.java:161); it never parses it again.

• Who. Code that turns node trees back into text to log, display, cache, forward or rewrite a query and then parses it again. The usual source is the syntax tree from StandardQueryParser.getSyntaxParser().parse(text, field), plus hand-built BoostQueryNode trees and processed trees with a configured field boost (setFieldsBoost(Map.of("title", 0.0005f)) prints title:lucene^5.0E-4). No option is needed.
• Trigger. A boost below 0.001 or above 2^63. Such boosts mostly come from code that computes them; a 2019 java-user post shows a client generating foo:bar^2.77E-4.
• Effect. Since 9.0.0 the printed query cannot be parsed again (ParseException), a loud failure. From 2.9.0 to 8.11.4 it silently re-parsed as a different query (f:ab^9 f:E-4). The fuzzy leg is silent on current releases.
• Not affected. Search results: the query is built from the exact float. Lucene's own code never re-parses the text.
• Likelihood: rare. It needs both the print-and-reuse pattern and an unusual boost; it is deterministic when both are present, and no matching public report was found in the recorded searches, although the code dates from 2.9.0 (2009).

The class sits in BoostQueryNode's package (org.apache.lucene.queryparser.flexible.core.nodes) so it can live next to the class upstream; it uses no package-private access. The seed only satisfies LuceneTestCase. It asserts only the documented round trip (the same boost after re-parsing), not an exact printed string.

Regression test: this same class (or its one method added to the existing TestQueryNode); it passes with the recommended fix.

cp $REPRO/TestBoostQueryNodeToQueryString.java $LUCENE/lucene/queryparser/src/test/org/apache/lucene/queryparser/flexible/core/nodes/
cd $LUCENE && ./gradlew :lucene:queryparser:test --tests TestBoostQueryNodeToQueryString -Ptests.seed=DEADBEEF -Ptests.asserts=true
# assertions off fails the same way: -Ptests.asserts=false

# gradle, stock 645d4fe14a20, -Ptests.seed=DEADBEEF -Ptests.asserts=true (stack trimmed; -Ptests.asserts=false identical)
TestBoostQueryNodeToQueryString > testSmallBoostParsesBack FAILED
    java.lang.AssertionError: expected text that parses back, got f:ab^9.0E-4
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:1EF405E732EBE7EA]:0)
        at org.apache.lucene.queryparser.flexible.core.nodes.TestBoostQueryNodeToQueryString.testSmallBoostParsesBack(TestBoostQueryNodeToQueryString.java:34)

        Caused by:
        INVALID_SYNTAX_CANNOT_PARSE: Syntax Error, cannot parse f:ab^9.0E-4:
            at app//org.apache.lucene.queryparser.flexible.standard.parser.StandardSyntaxParser.Boost(StandardSyntaxParser.java:1061)

# standalone runner, seed DEADBEEF: main 645d4fe14a20 fails with -ea and -da; 10.5.2 (-ea, -da) and 10.6.0 fail

# a 15-row boost test on stock main (gradle): printed text and whether it parses back
ab^0.001 -> f:ab^0.001 : ok
ab^0.5 -> f:ab^0.5 : ok
ab^0.0009999999 -> f:ab^9.999999E-4 : does not parse back (ParseException)
ab^0.00090 -> f:ab^9.0E-4 : does not parse back (ParseException)
ab^9223373000000000000 -> f:ab^9.223373E18 : does not parse back (ParseException)
ab^10000000000000000000 -> f:ab^1.0E19 : does not parse back (ParseException)
fieldsBoost 5.0E-4 -> title:lucene^5.0E-4 : does not parse back (ParseException)

# the same mistake in the fuzzy node, stock main (gradle)
java.lang.AssertionError: ab~0.0001 -> f:ab~1.0E-4: q1=f:ab~1 q2=f:ab~2 f:1.0e-4 expected:<0> but was:<1>

# before 9.0 (8.11.4, a public-API probe)
ab^0.0009 -> f:ab^9.0E-4 : parsed back as a different query: f:ab^9 f:E-4 (BooleanQueryNode)

• The contract: QueryNode.java:26 ("convert to a query string understood by the query parser"); core/nodes/package-info.java:42-43 ("A query node tree can also generate a query string that can be parsed back by the original text parser, at this point only the standard lucene syntax is supported."). Both date from 2.9.0.
• Upstream enforces it: GITHUB#7865, fixed by #14697 (merged 2025-05-29): "AbstractRangeQueryNode#toQueryString(EscapeQuerySyntax) now returns a string which can be parsed back into the original node" (lucene/CHANGES.txt:296-298, 11.0.0 bug fixes), with TestAbstractRangeQueryNode parsing the printed string back.
• Ordinary input: StandardQueryParser's javadoc documents "a floating-point weight boost" (jones^2 OR smith^0.5) with no lower bound, and 0.0009 is even inside the class javadoc's own (stale) range "0.0 to 1.0".
• The grammar was never meant to take exponents (a 2006 java-user reply about the classic parser: scientific notation would be a new feature; for the flexible grammar this is an inference), so the printer is the place to fix.
• Tests: no upstream test references BoostQueryNode or asserts its printed form.

The strongest case for "intended": in a 2017 java-user thread about range queries parsed twice, a committer replied: "There has never been a guarantee that going back and forth between a parsed query and its string representation is idempotent. so this isn't supported." Add that Float.toString is the normal Java rendering, that the grammar has no exponent on purpose, that Lucene uses toQueryString only for messages, that processor-only nodes print unparseable text ([TP[...]]), that the open #8667 (LUCENE-7616) calls the method "a design error" that "could never work as currently designed" (for custom syntaxes), and that BoostQuery.toString prints E-notation too.

Our answer, and its limits: in the same thread a PMC member replied "Why don't we fix this in Lucene? ... Could you open an issue and add a patch?", and the project later fixed exactly that range case as a bug (#7865 / #14697). The written contract says parse-back; Query.toString makes no such promise. #8667 is about custom syntaxes, while this is standard syntax to standard syntax, and StandardSyntaxParser produces BoostQueryNode from input it accepts. A grammar without exponents points at the printer as the fix site. This is a credible case for a bug, not a proof: #7865 concerned range nodes, so it is a precedent rather than a ruling on every node, and the processor-only nodes show the promise is not kept everywhere. Verifiers, the adjudication and a tie-break rejected the intended-behaviour case and a read-only cross-model review rated it weak; a later cross-model fact check rated the intended-behaviour case moderate for the boost leg, strong for boosts above 2^63 only, and weak for the fuzzy leg.

Two variants of the printer fix were written and validated in a scratch worktree of 645d4fe14a20; the grammar-side alternative is described but not run.

In BoostQueryNode.getValueString() keep the whole-number branch and Float.toString for Infinity and NaN, and otherwise print new BigDecimal(f.toString()).toPlainString(). In FuzzyQueryNode.toQueryString() print new BigDecimal(Float.toString(similarity)).toPlainString() for finite values in both return lines. +14/-3 in the two files, including the imports. The boost half alone fixes this repro; the fuzzy half fixes the merged fuzzy leg.

private CharSequence getValueString() {
  Float f = Float.valueOf(this.value);
  if (f == f.longValue()) return "" + f.longValue();
  // The grammar's NUMBER token has no exponent: print a plain decimal, never Float.toString's
  // E-notation (9.0E-4, 1.0E19), which does not parse back.
  else if (f.isInfinite() || f.isNaN()) return "" + f;
  else return new BigDecimal(f.toString()).toPlainString();
}

Trade-offs

• BigDecimal(Float.toString(f)) is the exact decimal Float.toString chose, so its plain form parses back to the identical float. Text that was already plain is unchanged (0.5, 2.0).
• Cosmetic: 0.0009 prints 0.00090 and the fuzzy ~0.0001 prints ~0.00010 (the trailing zero comes from 9.0E-4); 1e19 prints 10000000000000000000.
• Printing only; one BigDecimal per printed non-integral value. Parsing does not change.
• Visible output: toString() shares the helper, so <boost value='9.0E-4'> becomes <boost value='0.00090'>; exact-string tests, snapshots or log parsers that captured the old text see the new one (only for values that printed in E-notation). Very small values become long plain decimals: Float.MIN_VALUE prints as 48 characters (it still parses back).
• Out of scope: negative, NaN and infinite boosts. The grammar cannot produce them (no sign after ^) and BoostQuery rejects them (BoostQuery.java:40-41); a hand-built node prints f:ab^-0.5, f:ab^NaN or f:ab^Infinity, which does not parse back with or without the fix.

What the checks validated

• Gradle, scratch worktree of 645d4fe14a20, OpenJDK 25.0.3, seed DEADBEEF (2026-10-31): stock, the minimal test fails with assertions on and off, the 15-row boost test fails 2 of 2 and the fuzzy leg's test fails 2 of 2. With the fix: the minimal test passes with assertions on and off; the 15-row test passes and every control prints exactly as before; the fuzzy test passes 2 of 2; TestQueryNode passes; the whole queryparser module passes (30 classes, 533 tests, 0 failures); checkGoogleJavaFormat, ecjLintMain and forbiddenApisMain pass for the module.
• Earlier: a model of the boost fix had 0 failures over all 2,139,095,040 non-negative finite floats and caught a lossy mutant; the boost half passed the hunt's, a verifier's and the registration check's tests, and upstream TestQPHelper, TestAbstractRangeQueryNode and TestQueryNode (43 of 43).

new BigDecimal(f.toString()).stripTrailingZeros().toPlainString() in getValueString(), so ab^0.0009 prints ab^0.0009, the input back.

Trade-offs

Values already printed plainly are unchanged (a non-integral Float.toString result has no trailing zero). Do not use it in the fuzzy node: it would change today's ~2.0 to ~2.

What the checks validated

Same scratch worktree: the minimal test passes; the 15-row test passes and prints f:ab^0.0009, f:ab^0.0001 and title:lucene^0.0005; the whole queryparser module passes (533 tests, 0 failures).

Not run.

Trade-offs

Old printed text would parse, but it changes accepted syntax and how terms such as 1e5 are tokenized, for a bug in the printer. A 2006 list reply calls scientific notation a new feature.

Method: non-formal first signal (a differential round-trip probe), then bounded-exhaustive translation validation of the stock printer and a public-API JUnit.

• Signal. Code reading during the verification of LFV-892 (slop nodes print their int through a float) noted the lead; a planning probe (parse, toQueryString, re-parse, compare) executed it on stock main.
• Exhaustive check. All 2,139,095,040 non-negative finite floats through the real BoostQueryNode printer against a model of the NUMBER token (the model agreed with the real parser on 761,082 checks): exactly (0, 0.001) and (2^63, max] fail; 3,515 s on 4 threads. A host-construct grid through StandardQueryParser (19,620 checks) found only loud failures.
• Confirmation. An independent reproducer, a devil's-advocate verifier, read-only cross-model reviews (OpenAI GPT-5.6), an adjudication, the version-history runs, a prior-report search, a current-main reproducer and a tie-break checked it.