FunctionScoreQuery(q, DoubleValuesSource.fromLongField(f)) scores each match of q by the value of field f. Since #16431 it also reports each block's maximum score so that top-k search can skip blocks. With a TermQuery as q and a skip index on f, that maximum is 0.
TermQuery's 0, and after 1,000 hits the searcher skips every range whose bounds sum below 4.687542, including doc 4999's. Select the still to open it at full size. Generated with AI, checked against the code.- Index 5,000 documents that all contain
body:a; only doc 0 also containsbody:b. Apopularityfield with a skip index (NumericDocValuesField.indexedField) is 1 everywhere and 100 on doc 4999. - Search
BooleanQuery(SHOULD FunctionScoreQuery(TermQuery(body:a), fromLongField("popularity")), SHOULD TermQuery(body:b))withsearch(query, 1). You expect doc 4999 (score 100), which a search that scores every match returns. - You get doc 0 (score 4.69 = popularity 1 + 3.69 for
body:b). No error, with assertions on or off. - Why: the function query tells the searcher that no document in a block scores more than (wrapped query's max score) × (max popularity). It runs the
TermQuerywithout scores, so that max score is 0 and the claim is 0, although its documents score 1 to 100. - After 1,000 hits only scores above 4.69 can compete, and a clause that claims 0 cannot reach that. From doc 4096 on the searcher scores nothing that only the function clause matches, so doc 4999 is never looked at.
Applications on unreleased main that put a skip index on a ranking field (NumericDocValuesField.indexedField), wrap a TermQuery or similar in FunctionScoreQuery with fromLongField, fromIntField, fromFloatField, fromDoubleField or a monotonic fromField, and use it as a scoring clause of a BooleanQuery. boostByValue, boostByQuery and a top-level FunctionScoreQuery are not affected.
Mechanism, terms and numbers687 words
FunctionScoreQuery(q, DoubleValuesSource.fromLongField(f)) scores each match of q by the value of f; its score() is the value times the boost and never reads q's score. Since #16431 (main, 2026-08-22) its scorer also reports a per-block maximum score so that top-k search can skip blocks, computed as innerMaxScore * valueMaxScore * boost. Because the field source does not need scores, the wrapped query was weighted with COMPLETE_NO_SCORES, and a TermQuery weighted that way reports a max score of 0. So on a field with a doc-values skip index every block with a finite bound claims 0, and a BooleanQuery that uses the function query as a scoring clause can stop scoring the documents only that clause makes competitive: in the repro, search(query, 1) returns doc 0 (4.6875415) instead of doc 4999 (100). No released version is affected.
- Block-max pruning. For top-k search, each scoring clause reports, for a range of documents, the highest score any of them can have (
Scorer#getMaxScore). Once the collector knows the k-th best score so far, the searcher skips ranges whose bounds cannot beat it, without scoring their documents. - Skip index.
NumericDocValuesField.indexedField(name, value)adds a doc-values skip index: the minimum and maximum value per interval of documents. #16431 uses it to boundFunctionScoreQuery's scores per block. - Unscored wrapped query.
FunctionScoreQueryneeds its wrapped query only to pick the matching documents when the value source does not read scores, so it creates the wrapped query's weight withScoreMode.COMPLETE_NO_SCORES("scores won't be available").
createWeightweights the wrapped query withCOMPLETE_NO_SCORESunless the caller and the source both need scores (FunctionScoreQuery.java:109-117). Every field source (fromLongField,fromIntField,fromFloatField,fromDoubleField,fromField) does not need scores.- A
TermQueryweighted that way gets a dummySimScorerthat returns0f, "not expected to be called since scores are not needed" (TermQuery.java:77-91). Its max score is 0.0. score()returns(float) (value * boost)for a non-negative value, else 0 (FunctionScoreQuery.java:243-252). The wrapped score is never read.getMaxScore(upTo)returnsinnerMaxScore * valueMaxScore * boostwithinnerMaxScore = in.getMaxScore(upTo)(:262-269).valueMaxScoreis the skip index's maximum for the range (1.0 for docs 0-4998, 100.0 for doc 4999 in the repro), so the bound is 0.0 for both ranges, i.e. wherever the skip index gives a finite bound (a range past the current interval gets+Infinity). Without a skip index the value bound is+Infinityand the method returns+Infinity, which is why the skip index is part of the trigger.search(query, n)asks forTOP_SCORESfrom the start and counts the first 1,000 hits exactly; after that it raises the minimum competitive score to just above the lowest score in its top-n queue, the best score so far for n = 1 (IndexSearcher.java:100,TopScoreDocCollector.java:58-59,:154-170): 4.687542 after hit 1,001.MaxScoreBulkScorerthen treats every clause whose summed bounds stay below that as non-essential and only iterates essential clauses (MaxScoreBulkScorer.java:494-510). From doc 4096 on, the function clause claims 0.0 andbody:b, which has no documents left there, counts as 0 (:441-442); the sum, 0.0, is below 4.687542, so nothing in docs 4096-4999 is scored. The 4096 is the bulk scorer's inner window (INNER_WINDOW_SIZE = 1 << 12,:28): docs 0-4095 had been filled before the threshold was reached.- A top-level
FunctionScoreQueryalone is not pruned: itsFilterScorerinherits the no-opsetMinCompetitiveScore(Scorable.java:45-54). Its exposedgetMaxScoreis still 0.
An instrumented run of the repro index on main (assertions on and off) shows: doc 0 scores 1.0 + 3.6875415 = 4.6875415, docs 1-4998 score 1.0, doc 4999 scores 100.0; the function scorer's getMaxScore is 0.0 for docs 0-4998 and for doc 4999; setMinCompetitiveScore(4.687542) after 1,001 collected hits; collection stops after 4,096 documents; search(query, 1) returns doc 0, and search(query, 3) docs 0, 1 and 2, all with total hits "4096+". With the finder's fix the bounds are 1.0 and 100.0 and search(query, 1) returns doc 4999 while still counting only "4097+" hits, so pruning still skips docs 4096-4998 (not benchmarked).
Exposure and likelihood351 words
- Who is exposed. All of: unreleased main (11.0.0-SNAPSHOT) since #16431; a numeric doc-values field with a skip index; a score-independent field source with a finite bound; a wrapped query whose unscored max score is below 1; and the function query as a scoring clause of a
BooleanQueryunder the defaultsearch(query, n). - Wrapped queries. A
TermQueryreports 0. The finder's randomized differential also lost top hits with aConstantScoreQueryand withBooleanQuerydisjunctions and conjunctions of terms as the wrapped query.MatchAllDocsQueryreports 1 and is not affected. - Compositions. A
SHOULDclause next to another clause failed in every executed index shape. AMUSTclause next to another scoringMUSTclause failed in two independent tests, but whether it fails depends on how scoring windows line up with skip-index intervals. - Default or opt-in. The search side is all default (
IndexSearcher, BM25,search(query, n)). The repro opts into a skip index on the value field, a schema-time choice, and forces one segment (forceMerge(1)) to keep the index shape fixed. The skip index is exactly what #16431's speed-up relies on (CHANGES.txt: "Block-Max WAND dynamic score skipping in FunctionScoreQuery and DoubleValuesSource using DocValuesSkipper"), so the exposed applications are those that adopt the new feature as advertised and also use the query shape above. - Scale. A few thousand documents suffice (5,000 in the repro; 8,203 and 10,000 in the reviewers' tests); the exposed bound is 0 at any size for every finite skip-index bound. The values are ordinary non-negative integers.
- Not affected. A caller that needs no scores (then
createWeightreturns the wrapped weight itself,FunctionScoreQuery.java:118-120);boostByValueandboostByQuery(their source needs scores, so the bound stays+Infinity); a top-levelFunctionScoreQueryalone (right hits, wrong exposed bound); fields without a skip index; every released version. - What the damage looks like. Silent in the plain repro (only a test that wraps the inner query itself in
AssertingQueryfails an assertion): the returned hits are real matches with correct scores, but the best ones are missing, and the total hit count is the usual lower bound ("4096+").
The test shows that BooleanQuery(SHOULD FunctionScoreQuery(TermQuery(body:a), fromLongField("popularity")), SHOULD TermQuery(body:b)) over a skip-indexed popularity field (NumericDocValuesField.indexedField) returns the wrong top hit from the default IndexSearcher.search(query, 1). The cause is that the function clause's block-max bound is the wrapped no-score TermQuery's max score of 0 times the value bound.
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.lucene.queries.function;
import java.io.IOException;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.document.NumericDocValuesField;
import org.apache.lucene.document.StringField;
import org.apache.lucene.index.DirectoryReader;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.index.Term;
import org.apache.lucene.search.BooleanClause.Occur;
import org.apache.lucene.search.BooleanQuery;
import org.apache.lucene.search.DoubleValuesSource;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.Query;
import org.apache.lucene.search.ScoreDoc;
import org.apache.lucene.search.TermQuery;
import org.apache.lucene.store.ByteBuffersDirectory;
import org.apache.lucene.store.Directory;
import org.apache.lucene.tests.util.LuceneTestCase;
import org.apache.lucene.tests.util.TestUtil;
public class TestFunctionScoreQueryBooleanTopHit extends LuceneTestCase {
public void testTopHitWithSkipIndexedValueField() throws IOException {
try (Directory dir = new ByteBuffersDirectory();
IndexWriter w =
new IndexWriter(dir, new IndexWriterConfig().setCodec(TestUtil.getDefaultCodec()))) {
for (int i = 0; i < 4097; i++) {
Document doc = new Document();
doc.add(new StringField("body", "a", Field.Store.NO));
if (i == 0) {
doc.add(new StringField("body", "b", Field.Store.NO));
}
doc.add(NumericDocValuesField.indexedField("popularity", i == 4096 ? 100 : 1));
w.addDocument(doc);
}
try (DirectoryReader reader = DirectoryReader.open(w)) {
Query query =
new BooleanQuery.Builder()
.add(
new FunctionScoreQuery(
new TermQuery(new Term("body", "a")),
DoubleValuesSource.fromLongField("popularity")),
Occur.SHOULD)
.add(new TermQuery(new Term("body", "b")), Occur.SHOULD)
.build();
ScoreDoc top = new IndexSearcher(reader).search(query, 1).scoreDocs[0];
// expected doc 4096 (score 100 = its popularity); main returns doc 0 (score 1 + BM25 of b)
assertEquals("top hit (returned score " + top.score + ")", 4096, top.doc);
}
}
}
}
# in a copy of this finding's repro folder (repro/LFV-211/, which holds src/)
# Lucene checkout at the tested commit, built with: ./gradlew --no-daemon assemble (JDK 25)
LUCENE=/path/to/lucene; V=12.0.0-SNAPSHOT
CP=$LUCENE/lucene/core/build/libs/lucene-core-$V.jar
CP=$CP:$LUCENE/lucene/codecs/build/libs/lucene-codecs-$V.jar
CP=$CP:$LUCENE/lucene/test-framework/build/libs/lucene-test-framework-$V.jar
CP=$CP:$LUCENE/lucene/queries/build/libs/lucene-queries-$V.jar
CP=$CP:randomizedtesting-runner-2.9.2.jar:junit-4.13.2.jar:hamcrest-3.0.jar # test-framework dependencies (Maven Central)
CP=$CP:junit-platform-console-standalone-6.1.3.jar # JUnit Platform console launcher (Maven Central)
javac -d classes -cp "$CP" $(find src -name '*.java')
# run 1: assertions on, seed DEADBEEF
java -ea -esa --add-modules jdk.incubator.vector --enable-native-access=ALL-UNNAMED -Dtests.seed=DEADBEEF \
-cp "classes:$CP" org.junit.platform.console.ConsoleLauncher \
execute --select-class org.apache.lucene.queries.function.TestFunctionScoreQueryBooleanTopHit
- Expected vs actual
- Expected: doc 4096 (popularity 100, score 100). Actual on main with -ea and with -da: doc 0 (score 4.597 = 1 + BM25 of body:b),
expected:<4096> but was:<0>.
More files, environment, full output38 words
| lucene | Lucene main e1faeba08bde (2026-11-02) |
|---|---|
| jdk | 25.0.3+9-LTS |
# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
=> java.lang.AssertionError: top hit (returned score 4.5970817) expected:<4096> but was:<0>
org.junit.Assert.fail(Assert.java:89)
org.junit.Assert.failNotEquals(Assert.java:835)
org.junit.Assert.assertEquals(Assert.java:647)
org.apache.lucene.queries.function.TestFunctionScoreQueryBooleanTopHit.testTopHitWithSkipIndexedValueField(TestFunctionScoreQueryBooleanTopHit.java:67)score()never reads the wrapped score, yet the bound multiplies by it: 0.0 for documents scoring 1 to 100, against theScorer#getMaxScorecontract.- #16431's own test expects
maxScore >= 5000ffor this shape, but its field has no skip index, so the check never ran; made live, it fails. IndexSearcherpromises thatscoreDocs"is always accurate".
The full argument292 words
- Contracts.
Scorer#getMaxScore: "Return the maximum score that documents between the last target that this iterator was shallow-advanced to included and upTo included" (Scorer.java:75-79).Scorable#setMinCompetitiveScore: the scorer "may safely ignore all documents whose score is less than the given minScore" (Scorable.java:45-54); doc 4999 scores 100 and was ignored at 4.687542.IndexSearcher: "The TopDocs#scoreDocs array is always accurate however" (IndexSearcher.java:59-64).DoubleValues#getMaxScore: "the maximum score that documents ... can produce" (DoubleValues.java:55-61), i.e. the value bound already is the score bound. - The test framework enforces it.
AssertingScorerassertsscore <= getMaxScore(docID())(AssertingScorer.java:130),CheckHits.checkTopScoresassertsscore <= maxScore(CheckHits.java:766), andAssertingScorerasserts that scores are needed whenadvanceShalloworgetMaxScoreis called (:100,:116; LUCENE-8314), so asking the unscored wrapped scorer for bounds is itself off-contract. - Same-PR tests.
testMaxScoreDelegationWithDocValuesSkipper(>= 5000f),testMaxScoreMonotonicDecreasingFunction(>= 1000f) andtestCustomHomemadeIncreasingAndDecreasingFunction(>= 22.3f) all wrap aTermQueryand expect a bound of at least the largest value, but only when a skip index exists, and none of their fields has one (TestFunctionScoreQuery.java:385-420,:484-487,:583-586). With the field switched toindexedField(plusforceMerge(1)andupTo = maxDoc - 1), all three fail on main, e.g. "Expected maxScore >= 1000f but got 0.0", and pass with the fix. - Review history. The #16431 review (19 comments, 14 inline comments, 15 reviews) discussed value monotonicity, naming and constant functions; nobody mentioned the wrapped query's factor or its unscored weight. Before #16431 the bound was always
+Infinity, which is correct; 10.5.2 still returns the right hit. - Legal trigger.
NumericDocValuesField.indexedField, the field sources,FunctionScoreQueryandBooleanQueryare public and not experimental, and nothing documents a limitation.
The devil's-advocate check169 words
The strongest case for "intended": the product is deliberate. A comment in testConstantLambdaMonotonicity says the bound is "(5.0 * innerMaxScore)" (TestFunctionScoreQuery.java:705-707), and the unscored wrapped weight is deliberate too: it saves work and testScoreMode requires it (:303-323).
Why it fails: both choices can be deliberate and still not combine. score() never multiplies by the wrapped score, so with an unscored TermQuery the comment's own formula gives 5.0 * 0 = 0, below the 5.0 the test's documents score. Three assertions in the same PR expect a bound of at least the largest value with a TermQuery; they simply never ran. No javadoc, CHANGES or MIGRATE.md text restricts the query to a top-level position, to MatchAllDocsQuery, or to fields without a skip index, and the default search(query, 1) returns the wrong hit. Verdict: a defect (the cross-model review, the intended-behaviour check and the adjudication). Scope, not intent: a top-level function query returns the right hits, and the MUST + MUST case depends on window alignment.
Recommended A: do not consult the unscored wrapped scorer
When the value source does not need scores, stop consulting the unscored wrapped scorer: advanceShallow uses only the value source and getMaxScore uses a factor of 1 (+9/-2, FunctionScoreQuery.java). The repro, the finder's and reviewers' tests and the queries module pass (514 tests, 0 failures; six logged runs, assertions on and off); the trace still prunes.
final boolean innerScores = valueSource.needsScores();
// advanceShallow
int scoreShallow = scores.advanceShallow(target);
if (innerScores == false) {
return scoreShallow;
}
return Math.min(in.advanceShallow(target), scoreShallow);
// getMaxScore, after the +Infinity check
float innerMaxScore = innerScores ? in.getMaxScore(upTo) : 1f;
return innerMaxScore * valueMaxScore * boost;All options and trade-offs3 options · 644 words
Three variants were run; all fix the repro. They differ in what they do with the unscored wrapped scorer and with score-reading sources. A regression test for the PR is in the first option's notes. A over B is clear; A over C is a judgement about custom score-reading sources.
innerScores = valueSource.needsScores(); when false, advanceShallow returns the value source's boundary and getMaxScore multiplies by 1 instead of the wrapped scorer's max score. Score-reading sources (boostByValue, boostByQuery, DoubleValuesSource.SCORES) keep today's behaviour. Regression test: the test above as a new class, and six skip-index tests of #16431 in TestFunctionScoreQuery switched to a skip-indexed, single-segment field (NumericDocValuesField.indexedField, forceMerge(1); getMaxScore(maxDoc - 1) in five of them), which makes its three dead assertions live.
final boolean innerScores = valueSource.needsScores();
// advanceShallow
int scoreShallow = scores.advanceShallow(target);
if (innerScores == false) {
return scoreShallow;
}
return Math.min(in.advanceShallow(target), scoreShallow);
// getMaxScore, after the +Infinity check
float innerMaxScore = innerScores ? in.getMaxScore(upTo) : 1f;
return innerMaxScore * valueMaxScore * boost;Trade-offs
- The decision reads the rewritten source's
needsScores(), whilecreateWeightused the original source's; they agree for every source in Lucene, but passing the inner score mode into the weight would make the link explicit. - The live #16431 assertions need
forceMerge(1): they read only the first leaf, and with a random seed (513883939B3B8CD)newIndexWriterConfig()made several segments, so one failed even with the fix.upTo = maxDoclies past the last document and always yields+Infinitysince #16697, hencemaxDoc - 1.
What the checks validated
- Standalone harness, patched class on the stock jars: this repro 2/2 (assertions on and off), the finder's repro 3/3, the independent reproduction's tests 6/6 (on and off), the GPT-5.6 reviewers' tests 4/4 each, the intended-behaviour check's class 5/5 including its
AssertingQueryprobe. - gradle, scratch worktree of
ea57360912edwith the fix's test changes: without the fix 5 tests fail (this repro's 2, the 3 live #16431 assertions); with it:lucene:queries:testreports 514 tests, 0 failures (5-6 skipped) for seeds513883939B3B8CDandDEADBEEF(assertions on and off) and two random seeds: six logged runs (a seventh is recorded without its log; the fact check re-ran two more);:lucene:queries:check -x testpasses (google-java-format, ecjLint, forbiddenApis). - The diagram probe with the finder's variant (same bounds here): bounds 1.0 and 100.0;
search(query, 1)returns doc 4999 and still counts only "4097+" hits.
float innerMaxScore = valueSource.needsScores() ? in.getMaxScore(upTo) : 1f; and nothing else.
Trade-offs
advanceShallow still calls the unscored wrapped scorer, which AssertingScorer.advanceShallow rejects (assert canScore): the intended-behaviour check's AssertingQuery probe fails 1 of 5 with it. Only visible with assertions on and when the wrapped query is itself an AssertingQuery (newSearcher does not do that: the test above under newSearcher passes 10 of 10 with this option), but a reviewer will likely ask for the first option.
What the checks validated
The finder's repro 3/3, the independent reproducer's tests through an equivalent one-line variant of the change (its root-cause shadow), the GPT-5.6 reviewers' tests, upstream TestFunctionScoreQuery 17/17 and TestDoubleValuesSource 11/11; the registry re-run 3/3.
advanceShallow returns scores.advanceShallow(target) and getMaxScore returns valueMaxScore * boost, following score() and the DoubleValues#getMaxScore javadoc literally.
Trade-offs
For a score-reading source with a finite bound it trusts the source's own bound instead of multiplying. No source in Lucene reports a finite bound while needing scores (only FieldValuesSource computes a finite bound in core, and it needs no scores; DoubleValues.withDefault only delegates), so the change is invisible today but is a semantic decision for custom sources.
What the checks validated
This repro 2/2, the intended-behaviour check's class 5/5, :lucene:queries:test 514/514 for two seeds (assertions on and off).
An AI tool formed the hypothesis while modelling FunctionScoreQuery's bound: the wrapped factor in getMaxScore has no counterpart in score(), and the wrapped weight is unscored. An SMT check confirmed the arithmetic, a randomized top-k-versus-exhaustive differential found 3,946 mismatching query configurations, and a public-API JUnit reproduced it.
The method in full207 words
Method: semi-formal (a hunt unit, function queries and block-max pruning, 2026-10-26).
- Model. Writing the model's file:line table for
FunctionScoreQuery's scorer showed thatgetMaxScoremultiplies byin.getMaxScore(upTo)whilescore()never reads the wrapped score, and thatcreateWeightweights the wrapped query without scores for field sources. - SMT. z3 4.12.2 and cvc5 1.4.2 (floating-point theory; versions as declared by the hunt and printed by the installed solvers) found the bound unsound when the wrapped max score is 0 (query Q1a: sat). The 0 was an input taken from reading
TermQuery, so this step checked the arithmetic, not where the 0 comes from. - Differential. A randomized harness compared
TOP_SCORESwith exhaustive top-k for k = 1, 10, 50 and checkedscore <= getMaxScoreover random indexes: 3,946 mismatching top-k query configurations on stock main (450 complete indexes), 554 with this fix alone (450); the remaining mismatches have other causes; one job each of the first two levels was cut by the job timeout; none without a skip index or forboostByValue/boostByQuery. - Repro. A directed public-API JUnit then failed on the stock main jars with assertions on and off. A cross-model refute review (OpenAI GPT-5.6) reproduced it with its own tests before the candidate was written.
No prior report found (2026-10-26): 165 logged queries over Lucene issues, PRs, commits, mailing lists, JIRA, Elasticsearch, OpenSearch, Solr and the web, plus a full read of every apache/lucene issue, PR and comment since the code was written; 23 hits read, none is this bug. No open PR touches the code.
Searches5 searches
The code exists only since 2026-08-22, so the full read of GitHub activity since then covers every place a GitHub report could be. The only on-topic items are the feature request #16429, the introducing PR #16431 and the follow-up #16697; none mentions the wrapped query's factor.
| Where | When | How | Found |
|---|---|---|---|
| GitHub issues, PRs and commits (apache/lucene); Elasticsearch, OpenSearch, Solr | 2026-10-26 | a scripted prior-report search in three passes (class and method names; symptoms and trigger API such as indexedField, fromLongField, "getMaxScore returns 0"; the composition angle such as MaxScoreBulkScorer, setMinCompetitiveScore, non-scoring wrapped query) plus manual searches | no report; on-topic only #16429, #16431, #16697 |
| GitHub, full read through the API | 2026-10-26 | all 570 apache/lucene issues and PRs updated since 2026-08-15, all 1,038 issue comments and 689 review comments since 2026-08-20, the changed files of all 167 open PRs updated since 2026-08-22 | no report; no open PR touches FunctionScoreQuery.java, DoubleValuesSource.java, DoubleValues.java or their tests |
| PR conversations | 2026-10-26 | #16429 (body, 3 comments), #16431 (body, 19 comments, 14 inline comments, 15 reviews), #16697 (body, 2 comments) | review covered monotonicity, naming and constant functions; nothing on the wrapped factor |
| Mailing lists (dev@, java-user@, issues@, commits@) | 2026-10-26 | FunctionScoreQuery, getMaxScore, DoubleValuesSource, boostByValue, block-max over 2026-07-15 to 2026-10-27 | only notification mirrors of the on-topic items and unrelated items |
| JIRA archive and the web | 2026-10-26 | the tool's JIRA queries; 21 web queries | context only: LUCENE-9393 (#10433), #12297/#12383 (the 0 dummy scorer), LUCENE-8314 (#9361) |
Introduced on main by #16431 (0d4f93a9ff9, 2026-08-22), which replaced the +Infinity bound with the product; the wrong top hit is visible end to end since #16697 (d150ba27d11, 2026-10-09), which fixed an exception in front of it. No release is affected; still present on main a9779ef3c193 (2026-10-26).
Versions tested6 rows
Introduced by apache/lucene#16431 "Enable Block-Max WAND dynamic pruning for FunctionScoreQuery via DocValuesSkipper (+422% QPS)" (0d4f93a9ff9, by @rajat315315 with @uschindler as co-author, related issue #16429, merged 2026-08-22). The unscored wrapped weight is older (LUCENE-7623, 2017, which created the class; its ScoreMode form from LUCENE-4100, 2017; the current lines from LUCENE-9393, 2020), and so is the 0 max score of an unscored TermQuery (#12383, 2023); neither is a bug on its own. From the introducing commit until #16697 (d150ba27d11, by @easyice, 2026-10-09) the tested SHOULD query threw inside the skipper first (a separate advanceShallow defect that #16697 fixed), although the bound was already 0; other query shapes in that window were not run. CHANGES.txt lists both under Lucene 11.0.0; branch_10x, branch_10_6 and branch_10_5 do not contain #16431. The commit-level runs are plain javac builds on the class path, not gradle builds; at d150ba27d11 they match the gradle-built main jars message for message.
| Version or commit | Verdict | Result |
|---|---|---|
main ea57360912ed, a9779ef3c193 | affected | wrong top hit and bound 0.0 (ea57360912ed: assertions on and off; a9779ef3c193: the finder's test, assertions on) |
d150ba27d11 (#16697) | affected | same messages as main |
0d4f93a9ff9 (#16431) | introducing commit | bound 0.0; the tested query threw in the skipper (AssertionError with -ea, EOFException with -da); with #16697's change shadowed on top, the silent wrong hit appears, and the fix makes it pass |
a1fcd50799f (parent of #16431) | bisect | bound +Infinity, right top hit |
| 10.0.0, 10.5.2 | not affected | bound +Infinity, right top hit |
| 9.12.3 | not applicable | no NumericDocValuesField.indexedField (compile error) |
An independent reproducer, a devil's advocate and OpenAI GPT-5.6 reviewers each reproduced it on main with their own tests and found the "intended" case weak; an adjudication re-ran the independent reproduction's tests and the finder's repro and confirmed it with high confidence. This doc's fix validation found that the live #16431 assertions need forceMerge(1).
Checks and verdicts383 words
Every check ran its own code on the stock main jars; all controls (no skip index, MatchAllDocsQuery as the wrapped query, the function query alone) passed.
| Check | Verdict | What it found |
|---|---|---|
| Finder (a hunt unit) | candidate | SMT check and differential; the directed JUnit fails 3/3 on main with assertions on and off; its fix passes 3/3. |
| Refute review (OpenAI GPT-5.6, executed) | real (high) | Own tests on 137 and 8,203 documents: "bound 0.0 must cover actual score 146.0", and a MUST conjunction whose top hit is doc 0 instead of doc 8202; controls and 10.5.2 pass; the fix passes. |
| Independent reproducer | real | Own tests on 10,000 documents, default codec pinned: 4 of 6 fail with assertions on and off (bound, CheckHits, SHOULD, MUST + MUST); 20 of 30 over 5 random-seed iterations; 10.5.2 6/6 pass. |
| Intended-behaviour review (OpenAI GPT-5.6, executed) | real (high); intended weak | Own bound and default-API tests fail on main (search(query, 1) returns doc 0 instead of doc 4999); re-run separately with the same results. |
| Devil's advocate | real | No intended, documented or misuse argument survives; three #16431 assertions made live fail on main and pass with the fix. |
| Adjudication | confirmed (high) | Re-ran the independent reproduction's tests with a fresh seed and assertions off (4 of 6 fail) and the finder's repro (3/3 fail); corrected the MUST + MUST wording. |
| Follow-up and registry re-run | new; security none | History sweep (25 runs), prior-report search, security assessment; fresh re-runs fail 3/3 on main and pass with the fix; also reproduces on main a9779ef3c193. |
| This doc's fix validation | fix validated; test corrected | The recommended fix passes the queries module (514 tests, 0 failures) in six logged runs, plus a seventh recorded without its log; the live #16431 assertions failed once with the fix under a multi-segment random seed until forceMerge(1) was added. |
| Fact checks and diagram check | text corrected; bug confirmed | A Claude fact check re-ran the repro through gradle and the runner and found two text errors (body:b's bound after doc 4096 is 0, not 3.6875415; the reason given for avoiding newSearcher was wrong). An OpenAI GPT-5.6 fact check found no error that changes the conclusion. The diagram check passed. All corrections are on this page. |
The diagram as a still image, generated with AI and checked against the code. It is under 10 MB, the limit for a GitHub attachment.
- PNG 622 KB
Finding record, Lucene and upstream sources, repro files and diagram notes1,455 words
- Registry title
- FunctionScoreQuery's block-max bound multiplies by the wrapped query's no-score max score (0 for a TermQuery), so a BooleanQuery that contains it can drop the true top hits
- Area
- queries / FunctionScoreQuery block-max pruning (#16431)
- Priority
- Medium, rank 102: On main since #16431 FunctionScoreQuery's block-max bound multiplies by the wrapped TermQuery's no-score max of 0, so a BooleanQuery with a skip-indexed function clause silently loses its true top hits with ordinary values, the new feature's main use case, to fix before 11.0.
- Impact class
- silent wrong results
- Impact note
- Applications on unreleased main (11.0.0-SNAPSHOT, since #16431) that build FunctionScoreQuery(q, fromLongField, fromIntField, fromFloatField, fromDoubleField or fromField with INCREASING/DECREASING) around a wrapped query whose no-score max score is below 1 (TermQuery reports 0; a ConstantScoreQuery or BooleanQuery of such clauses too), on a numeric field indexed with the opt-in doc-values skip index (NumericDocValuesField.indexedField), and use it as a scoring clause of a BooleanQuery: the default IndexSearcher.search(q, n) silently returns wrong top hits (5,000 docs: doc 0 with score 4.69 instead of doc 4999 with score 100), with the default codec and searcher, -ea or -da, at a few thousand documents and with ordinary non-negative values; a top-level FunctionScoreQuery alone still returns the right hits (only its exposed getMaxScore is 0), MatchAllDocsQuery as the wrapped query and the boostByValue/boostByQuery helpers are not affected, and no released version is affected.
- Summary
- You index a popularity field with a skip index (NumericDocValuesField.indexedField("popularity", v)) and search BooleanQuery(SHOULD FunctionScoreQuery(TermQuery(body:a), DoubleValuesSource.fromLongField("popularity")), SHOULD TermQuery(body:b)) for the top hit, expecting the same hit as an exhaustive search, because the function query scores each matching document by its popularity value. Instead, with 5,000 documents that all match body:a, where only doc 4999 has popularity 100 (all others 1) and only doc 0 matches body:b, TOP_SCORES and the plain IndexSearcher.search(q, 1) return doc 0 (score 4.69) instead of doc 4999 (score 100). The cause is that the scorer's getMaxScore returns innerMaxScore * valueMaxScore * boost, where innerMaxScore is the max score of the wrapped TermQuery, which FunctionScoreQuery itself weighted without scores (so it reports 0), although score() is value * boost and never reads the wrapped score; every block therefore reports a bound of 0, and MaxScoreBulkScorer stops scoring documents that only the function clause makes competitive.
- Introduced
- 0d4f93a9ff93ccc396c1323f26416b57d8dc08d9 on main (apache/lucene#16431 'Enable Block-Max WAND dynamic pruning for FunctionScoreQuery via DocValuesSkipper (+422% QPS)', by rajat315315 with uschindler as co-author, related issue #16429, merged 2026-08-22) [exec: the follow-up history sweep shows the 0.0 bound at 0d4f93a9ff9, d150ba27d11 and main with -ea and -da, while the parent a1fcd50799f, 10.0.0 and 10.5.2 report +Infinity and return the right top hit; read: git log -S innerMaxScore on FunctionScoreQuery.java returns only 0d4f93a9ff9, and git blame of FunctionScoreQuery.java:254-268 is that commit]; it replaced the scorer's +Infinity bound with innerMaxScore * valueMaxScore * boost, where the inner scorer comes from the COMPLETE_NO_SCORES weight that createWeight has used for score-independent sources since LUCENE-4100 (4fc5a872ded, 2017; the current lines 111-117 are from LUCENE-9393, e1a97a0f1ea, 2020), and a TermQuery weighted that way has reported a max score of 0 since #12383 (40ee6e583e1, 2023, a dummy no-score SimScorer); the silent wrong top hit became reachable end to end with #16697 (d150ba27d11, by easyice, 2026-10-09), which fixed a separate advanceShallow defect that made the tested SHOULD query throw first [exec: #16697's DoubleValuesSource change shadowed onto 0d4f93a9ff9 gives main's wrong top hit, and adding the fix makes all 3 tests pass] (2026-08-22 (0d4f93a9ff9 on main); silent wrong top hit reachable end to end from 2026-10-09 (d150ba27d11, #16697) for the executed SHOULD query shape)
- First affected version
- unreleased (main only; 11.0.0 would be the first release with it) [exec: 10.0.0 and 10.5.2 correct; read: no release tag contains 0d4f93a9ff9, and branch_10x, branch_10_6 and branch_10_5 do not contain #16431]
- Affected versions
- main (11.0.0-SNAPSHOT) from 0d4f93a9ff9 (2026-08-22) through 645d4fe14a2 (upstream main head, re-run 2026-10-29) [exec: Scorer#getMaxScore 0.0 at 0d4f93a9ff9, d150ba27d11 and main, -ea and -da; silently wrong TOP_SCORES top hit at d150ba27d11 and main, and at 0d4f93a9ff9 only with #16697's change shadowed, because before #16697 the tested SHOULD query threw in the skipper first (AssertionError with -ea, EOFException with -da)]; not affected: 10.0.0 [exec, -ea] and 10.5.2 [exec, -ea and -da] (FunctionScoreQuery's bound is +Infinity there) and the parent commit a1fcd50799f [exec]; not applicable: 9.12.3 [exec: compile error, no NumericDocValuesField.indexedField; the bound is +Infinity, read]; unreleased branch_10x, branch_10_6 and branch_10_5 do not contain #16431 [read; exec 2026-10-29 on the branch_10_6 RC1 4ae946e5a9b snapshot jars: not affected, -ea and -da]
- Status
- documented (brief, checks and a per-finding doc exist)
- Verification
- Confirmed by AI-run checks: an independent repro, a cross-model review that agreed, an intended-behaviour check, a prior-report search and a skeptical audit; not yet triaged by a person.
- Found by
- semi-formal: code reading; also reached by Z3/cvc5, fuzzing/PBT
- Upstream and prior reports
- none found (2026-10-26, the prior-report record: 165 logged read-only queries, 121 through a scripted prior-report search over apache/lucene issues and PRs, commits, the dev@, java-user@, issues@ and commits@ lists, the LUCENE JIRA archive, the Elasticsearch, OpenSearch and Solr repositories and the web, and 44 manual ones; plus a full read through the GitHub core API of all 570 apache/lucene issues and PRs updated since 2026-08-15, all 1,038 issue comments and 689 review comments since 2026-08-20, and the changed-file lists of all 167 open PRs updated since 2026-08-22; 23 hits read, none of them this bug; the finder's scripted prior-report search and manual searches and the independent reproduction's and the intended-behaviour check's searches found nothing either); not fixed on upstream main (HEAD ea57360912ed on 2026-10-26: FunctionScoreQuery.java is byte-identical to the tested worktree and getMaxScore still returns innerMaxScore * valueMaxScore * boost; no open PR touches FunctionScoreQuery.java, DoubleValuesSource.java, DoubleValues.java or their tests); origin: #16431 (related issue #16429; its review discussed value-function monotonicity, naming and constant functions, and nobody mentioned the innerMaxScore factor or the COMPLETE_NO_SCORES inner weight); #16697 (easyice) fixed other defects of the same feature in DoubleValuesSource and does not touch FunctionScoreQuery; context, not reports: #10433 (LUCENE-9393, the inner weight's score mode), #12297 and #12383 (TermQuery's no-score dummy scorer that returns 0), #9361 (LUCENE-8314, AssertingScorer checks that scores are needed when advanceShallow and getMaxScore are called)
- Lucene commit tested
- main
ea57360912ed(2026-10-24; tested 2026-10-26), seed DEADBEEF and random seeds, assertions on and off, through gradle and through a standalone JUnit harness; maina9779ef3c193(2026-10-26, sameFunctionScoreQuery.java); 10.5.2, 10.0.0 and 9.12.3 jars; the introducing commit0d4f93a9ff9, its parent andd150ba27d11(commit-level builds); the fix in a scratch worktree ofea57360912ed.
- Finding record
a Lucene main checkout lucene/queries/src/java/org/apache/lucene/queries/function/FunctionScoreQuery.java:84-85 (boostByValue uses MultiplicativeBoostValuesSource, needsScores true at :310), :111-117 (createWeight weights the wrapped query with COMPLETE_NO_SCORES unless both the caller and the source need scores), :243-252 (score() returns (float) (value * boost), never the wrapped score), :254-259 (advanceShallow), :262-268 (getMaxScore returns innerMaxScore * valueMaxScore * boost)a Lucene main checkout lucene/core/src/java/org/apache/lucene/search/TermQuery.java:75-92 (no-score TermWeight: dummy SimScorer returning 0f, 'not expected to be called since scores are not needed'); lucene/core/src/java/org/apache/lucene/search/DoubleValuesSource.java (FieldValuesSource.getMaxScore returns the decoded skipper bound; needsScores false)a Lucene main checkout lucene/core/src/java/org/apache/lucene/search/Scorer.java:75-79 (getMaxScore contract), Scorable.java:44-53 (setMinCompetitiveScore: may ignore documents scoring below minScore; no-op by default, which FilterScorer inherits), MaxScoreBulkScorer.java:498-504 (clauses become non-essential while the summed bounds stay below minCompetitiveScore), BlockMaxConjunctionBulkScorer.java:160-166 (a window whose bound is below minCompetitiveScore is skipped), IndexSearcher.java:100 (TOTAL_HITS_THRESHOLD = 1000)a Lucene main checkout lucene/queries/src/test/org/apache/lucene/queries/function/TestFunctionScoreQuery.java:385-417 (testMaxScoreDelegationWithDocValuesSkipper: 'assertTrue(maxScore >= 5000f)' only inside 'if (getDocValuesSkipper("val") != null)', on a field without a skip index), :635 and :707 (testConstantLambdaMonotonicity comment '(5.0 * innerMaxScore)'); lucene/test-framework/src/java/org/apache/lucene/tests/search/CheckHits.java:766 and AssertingScorer.java:100 and :130 (canScore and score <= maxScore oracles); lucene/CHANGES.txt:223 (Lucene 11.0.0: GITHUB#16431, GITHUB#16697)
- The finding in 30 seconds
Lucene main ea57360912ed: lucene/queries/src/java/org/apache/lucene/queries/function/FunctionScoreQuery.java:109-117 (createWeight), :241 (FilterScorer), :243-252 (score), :262-269 (getMaxScore)Lucene main ea57360912ed: lucene/core/src/java/org/apache/lucene/search/TermQuery.java:77-91; DoubleValuesSource.java:572-584 and :603-606; ScoreMode.java:25-26Lucene main ea57360912ed: lucene/core/src/java/org/apache/lucene/search/IndexSearcher.java:100, :598-599; TopScoreDocCollector.java:154-170; TopScoreDocCollector.java:58-59; MaxScoreBulkScorer.java:28, :36-39, :441-442, :487-493, :494-510; Scorable.java:45-54
- Who is affected
Lucene main ea57360912ed: lucene/queries/src/java/org/apache/lucene/queries/function/FunctionScoreQuery.java:84-86, :309-312 (MultiplicativeBoostValuesSource needs scores)Lucene main ea57360912ed: lucene/CHANGES.txt:223-224 (Lucene 11.0.0)
- Why it looks unintended
Lucene main ea57360912ed: lucene/core/src/java/org/apache/lucene/search/Scorer.java:75-79; Scorable.java:45-54; IndexSearcher.java:59-64; DoubleValues.java:55-61Lucene main ea57360912ed: lucene/test-framework/src/java/org/apache/lucene/tests/search/AssertingScorer.java:100, :116, :130; CheckHits.java:766Lucene main ea57360912ed: lucene/queries/src/test/org/apache/lucene/queries/function/TestFunctionScoreQuery.java:303-323, :385-420, :484-487, :583-586, :705-707
- Suggested fix
Lucene main ea57360912ed: lucene/core/src/java/org/apache/lucene/search/DoubleValuesSource.java:556-569 (getLevel), :572-584
- History
Lucene main ea57360912ed: lucene/CHANGES.txt:223-224
- TestFunctionScoreQueryBooleanTopHit.java (JUnit test):
repro/LFV-211/src/org/apache/lucene/queries/function/TestFunctionScoreQueryBooleanTopHit.java(71 lines)
A picture of the 5,000 documents along a doc-id axis with the two clauses, the skip index's ranges, the collector's threshold after 1,000 hits and the windows MaxScoreBulkScorer scores or skips; every number is taken from the instrumented run of the repro index. The PNG is the final summary card.
assets/LFV-211/bug.png(622 KB)
