ContextQuery gives an exact context's boost to longer indexed contexts that start with it

A ContextQuery in the document suggester decides which suggestions match with one rule and how much to boost them with another. Matching honours exact contexts: exact red admits only suggestions indexed under red. The boost lookup does not: it keeps only the boost of each query context, drops the exact flag, and takes the longest query context whose characters equal the start of the matched indexed context, without checking that the indexed context ends there. So when another clause of the same query, addAllContexts() or a shorter prefix context, lets a longer context such as redwood in, that suggestion gets the boost of exact red: table lamp (weight 5) scores 50 instead of 5 and ranks above table cloth (40). An exact empty context with addAllContexts() gives its boost to every suggestion with a context. No error is raised; the set of suggestions the context filter admits and the reported contexts are right; only scores are wrong, and so the order and which admitted suggestions make the top N. The code dates from LUCENE-6459 (2015); every release from 5.3.0 to 10.5.2 and main are affected.

• Document suggester: suggestions indexed with ContextSuggestField (key, weight, optional contexts) and a completion postings format, queried through SuggestIndexSearcher. A suggestion scores weight * boost; with boost 0 it scores its plain weight, and with weight 0 the boost (CompletionScorer.java:98-106).
• ContextQuery (@lucene.experimental) wraps a completion query: addContext(c, boost) adds an exact context, addContext(c, boost, false) a prefix context, addAllContexts() lets every context in "with a boost of 1f".
• In the index each suggestion path is context + CONTEXT_SEPARATOR (\u001D) + key; with the default preserveSep a SEP_LABEL (0x1F) follows the separator.

1. Filtering is right. toContextAutomaton puts the separator directly after an exact context, and any characters + separator after a prefix context or for addAllContexts() (ContextQuery.java:228-250, :241-244). Exact red alone never returns redwood.
2. The boost table forgets exact. createWeight stores only the boost per context: contextMap.put(entry.getKey(), contextMetaData.boost) (:208), and sorts the context lengths longest first (:211-212).
3. The lookup checks only a prefix. For each matched path prefix, setNextMatch cuts the path to each length, longest first, and takes the first key found (:285-300). For redwood\u001Dtable lamp and the query contexts red and re, length 3 gives red, so boost 10. Nothing checks that the separator follows at position 3.
4. The score. table lamp gets 5 x 10 = 50 instead of 5 x 1 = 5. Its reported context is still redwood, because setInnerWeight copies the whole indexed context (:308-331).
5. Empty exact context. A key of length 0 equals the start of every path, so exact "" (meant for suggestions without a context) boosts every path that no longer key matches. With only exact "" plus addAllContexts(), suggestions without a context get the boost rightly (table mat 30) and suggestions with a context wrongly (table lamp 50, table cloth 40).

Probe on main, key prefix tab; table lamp 5 under redwood, table cloth 4 under red, table mat 3 without a context:

It shows only when the stolen boost gives a different score from the right one (the boost of the longest query context that really matches, or none under addAllContexts()). The scorer sees the context boost plus the inner query's boost (ContextQuery.java:339-341). With a prefix inner query, as here, boosts 0 and 1 both leave the plain weight, so an exact boost of 1 or 0 is hidden under addAllContexts() or a boost-1 prefix context; a fuzzy inner query adds its own boost, so there exact red^1 with addAllContexts() shows (table lamp 15 instead of 10, ranked first). It is also hidden when a longer query context matches. The wrong boost can lower a score too (addContext accepts boost 0): prefix re^2 + exact red^0 gives table lamp 5 instead of 10.

• The pattern is upstream-tested. addAllContexts() plus boosted exact contexts is what TestContextQuery.testMixedContextQuery does (addAllContexts() plus exact type1^7 and type2^6). Index one more context, type10, and its suggestion (weight 5) scores 35 instead of 5.
• Ordinary names trigger it. Category sets where one name starts with another: numbered or hierarchical names, numeric ids.
• Downstream (code reading only, not run): Elasticsearch and OpenSearch build ContextQuery from category contexts with addContext(typeId + context, boost, !isPrefix) and require at least one context, so they never call addAllContexts(); there it takes one request that mixes an exact category and a shorter prefix category whose boosts give different scores.
• What it costs users. Silent wrong scores (the stolen boost replaces the right one), so a wrong order, and an admitted suggestion can be pushed below the top N (with num = 1 the example returns table lamp instead of table cloth). The set of suggestions the query admits and each hit's reported context do not change.
• What it takes. The document suggester and ContextQuery are opt-in; nothing else is: the default codec with the completion format, assertions on or off, no scale.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: score of lamp/redwood expected:<5.0> but was:<50.0>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:577)
       org.apache.lucene.search.suggest.document.TestContextQueryExactContextBoost.testExactContextBoostNotAppliedToLongerContext(TestContextQueryExactContextBoost.java:50)

• Contracts (ContextQuery.java): "Adds an exact context with boost" (:115); "set exact to false if the context is a prefix of any indexed contexts" (:120-123); class javadoc: "To apply the same boost to multiple contexts sharing the same prefix, Use addContext(CharSequence, float, boolean) with the common context prefix, boost and set exact to false" (:64-66); addAllContexts(): "Add all contexts with a boost of 1f" (:146); ContextMetaData.exact: "whether the context value should be treated as an exact value or a context prefix" (:256-257).
• Upstream tests. testMixedContextQuery (TestContextQuery.java:311) and testEmptyContextWithBoosts (:236) expect contexts without a configured boost to keep their plain weight under addAllContexts(). testContextPrecedenceBoost (:177) gives each indexed context the boost of the query context equal to it, and queries exact type while typetype is indexed, so that usage is legal. No upstream test indexes a longer context that an exact query context starts with and that another clause lets in; many tests use boost 1, which hides the bug.
• History. The lookup came with ContextQuery in LUCENE-6459 (2015). Its only recorded boost intent is "added test to ensure longer context takes precedence for boosting", i.e. testContextPrecedenceBoost. No issue, CHANGES or MIGRATE text calls the behaviour intended.
• Under any reading that respects exact, redwood is matched only by re^1 or by addAllContexts(), so 50 is wrong whatever tie-break among matching contexts is chosen.

The intended-behaviour checker built the strongest case: (a) the lookup is deliberately "longer context first" (code comment, LUCENE-6459's precedence test), so the longest query context that starts the indexed context supplies the boost whatever its flag; (b) read literally, "set exact to false if the context is a prefix of any indexed contexts" is a precondition, so exact red with redwood indexed is misuse; (c) the class is @lucene.experimental; (d) the code can be read as admission by the automaton, then a boost overlay by string prefix, unchanged since 2015.

Why it fails: (a) the precedence rule picks among query contexts that match, and the fix keeps it (the controls with prefix re^2 + exact redwood^3 and with two prefix contexts pass with and without the fix); (b) upstream's own testContextPrecedenceBoost queries exact type with typetype indexed, and filtering honours exactness in that case; (c) experimental allows API changes, not wrong scores; (d) nothing documents a prefix overlay for exact contexts, and the age of the behaviour argues for a release note, not for keeping the scores. The literal reading (b) is the strongest point; a maintainer who prefers it would document the behaviour instead (option D). Verdict: a defect. Not security: only ranking within the admitted set changes.

One fix was built and validated in a scratch worktree at ea57360912ed; the other options were not run.

createWeight stores ContextMetaData (boost and exact) in contextMap instead of the boost alone; setNextMatch accepts a key of length N that is exact only when the path has CONTEXT_SEPARATOR at position N. +12/-6 lines in ContextQuery.java; contextMap and ContextCompletionWeight are private, so no API change. The finder wrote it, and the independent reproducer wrote the same fix on its own.

// createWeight: keep the exact flag (contextMap is now Map<IntsRef, ContextMetaData>)
contextMap.put(entry.getKey(), contextMetaData); // was: contextMetaData.boost

// setNextMatch: an exact context only matches when the indexed context ends right here
ContextMetaData metaData = contextMap.get(ref);
if (metaData != null
    && (metaData.exact == false
        || (contextLength < pathPrefix.length
            && pathPrefix.ints[pathPrefix.offset + contextLength]
                == ContextSuggestField.CONTEXT_SEPARATOR))) {
  currentBoost = metaData.boost;

Trade-offs

• Cost. The map is probed once per context length tried, as before, and the second lookup on a hit (containsKey then get) is gone; an exact key that is found adds a bounds check and one array read. The loop runs once per matched path prefix, not per document. No benchmark was run.
• Behaviour change only where an exact query context is a proper prefix of the matched indexed context: that path now gets the boost of the longest query context that really matches it, or none (plain weight). Exact keys equal to the indexed context, and all prefix keys, behave as before.
• The exact empty context then boosts only suggestions without a context, as its filtering already does.
• Ranking change. Affected queries get different scores and order; the behaviour dates from 2015, so a CHANGES.txt entry is advisable.

What the checks validated

• Stock: the test fails 3 of 5 with assertions on and off; both controls pass.
• Fix: the test and the closest suites (TestContextQuery 15, TestContextSuggestField 6, TestFuzzyCompletionQuery 3, the new test 5) pass 29 of 29, assertions on and off.
• Fix, :lucene:suggest:test: 32 classes, 241 tests, 0 failures, 7 skipped (@Nightly and benchmark tests), assertions on and off, and with a second seed.
• :lucene:suggest:tidy left the fix unchanged; :lucene:suggest:check -x test passes (google-java-format, ecjLint, forbiddenApis).
• Also passes the finder's, the independent reproducer's and the devil's advocate's tests. The random harness, run with this fix and the separate NRTSuggester fix (LFV-242) shadowed in, finds no mismatch of this class on 3,000 indexes with deletions where main has 8,808 (a combined-fix run; the tests above isolate this fix).

Find the separator once (the indexed context is the path up to it), look up the whole context with any flag, and otherwise take the longest shorter key whose exact is false.

Trade-offs

Same results as A, since an exact key can only equal the whole context; states the intent more directly, at the cost of reshaping the loop.

Store each exact key with CONTEXT_SEPARATOR appended, so a plain prefix comparison can only match where the indexed context ends.

Trade-offs

No flag check in the loop, but the matched length then includes the separator, so the offset handed to setInnerWeight needs adjusting; more invasive than A.

State that an exact context's boost also applies to longer contexts that start with it.

Trade-offs

Contradicts "Adds an exact context", the class javadoc's advice to use exact=false for a shared prefix boost, and the filter, which treats the same context as exact in the same query.

A hunt unit (suggesters). The harness compares the stock suggester with the reference on random indexes of at most 6 documents over an alphabet of a, b and space, contexts from {a, b, ab, aa, ba, aba}, random exact and prefix query contexts with boosts and addAllContexts(), every key of length up to 3, 1 to 5 results, duplicates skipped or kept. On main, 3,000 indexes without deletions gave 17,310 mismatches, all "exact context boost applied to a longer context"; on 3,000 indexes with deletions, 8,808 of this class on main and none with this fix and the separate NRTSuggester fix (LFV-242) shadowed in (a combined-fix run; the JUnit and probe runs isolate this fix). The harness samples randomly and is not exhaustive; no model checker or prover was involved.

Then: a cross-model refute review (an independent reviewer from another model family, OpenAI GPT-5.6, reading only), an independent reproducer, a second GPT-5.6 review arguing "works as designed", a devil's advocate, an adjudication, a release sweep from 5.2.1 to main, a prior-report search and a security assessment.