FuzzyCompletionQuery with unicodeAware=true compares the query's code points with the suggestion's UTF-8 bytes, so its prefix boost stops at the first non-ASCII letter

FuzzyCompletionQuery, the document suggester's fuzzy prefix query, boosts each suggestion by the number of leading letters its matched prefix shares with the typed text, so an exact prefix normally gets a larger boost than a typo (a typo in the last letter ties with it, the minimal-prefix NOTE). With unicodeAware=true, documented as measuring the query "in Unicode code points (actual letters) instead of bytes", that count compares two units: the query is kept as code points, the matched suggestion arrives as UTF-8 byte labels (FuzzyCompletionWeight.setNextMatch, FuzzyCompletionQuery.java:266-286). ASCII letters have the same value in both, so the count runs through ASCII letters and, when the comparison reaches the first non-ASCII query letter, stops there; an all-non-ASCII query gets boost 0, and a code point can accidentally equal an unrelated byte (U+00C3 and the first byte of é), which can even give a typo a larger boost than the exact prefix. With the default fuzzy settings, aéc over aécd/3 and the typo axcd/4 gives [axcd/4.0, aécd/3.0] instead of the ASCII twin's [abcd/6.0, axcd/4.0], and кот over коты/3 and киты/4 gives [киты/4.0, коты/3.0]. Which suggestions match is right; their scores and order change, and with them which matches make a top-n list, in the plain query and inside ContextQuery. It has been this way since the class was added (LUCENE-6459, first release 5.3.0). Decoding the matched path to code points before comparing (+15/-2, one file) fixes it, and the whole suggest module still passes.

• Document suggester: suggestions indexed as SuggestField (or ContextSuggestField) values, each with an integer weight, searched with SuggestIndexSearcher. Internally an FST whose labels are UTF-8 bytes.
• FuzzyCompletionQuery (@lucene.experimental): returns suggestions whose prefix is within maxEdits of the typed text (defaults: maxEdits 1, transpositions on, nonFuzzyPrefix 1, minFuzzyLength 3). Score = weight x boost; a boost of 0 gives the weight (CompletionScorer.java:98-106).
• Boost: for each matched prefix path, the longest run of leading letters it shares with the query (FuzzyCompletionWeight.setNextMatch, FuzzyCompletionQuery.java:266-286). The path is the shortest prefix the fuzzy automaton accepts, so for the query abc the exact suggestion abcd is scored on ab (the NOTE at :268-271): that is why the expected boosts are 2 and 1, not 3 and 1.
• unicodeAware (default false, :61): "Measure maxEdits, minFuzzyLength, transpositions and nonFuzzyPrefix parameters in Unicode code points (actual letters) instead of bytes" (:57-59).

1. createWeight analyzes the query with stream.toAutomaton(unicodeAware) (:150-153); in unicode-aware mode its labels are code points. toLevenshteinAutomata keeps every string of it as a boost reference, refs.add(IntsRef.deepCopyOf(string)) (:166-170).
2. Because the FST has byte labels, only the match automaton is converted to UTF-8 with UTF32ToUTF8 (:156-160); the refs are passed on unchanged (:163).
3. NRTSuggester calls setNextMatch(path.input().get()) with the matched path in UTF-8 bytes (NRTSuggester.java:230-231); ContextQuery forwards the bytes after the context (ContextQuery.java:308-328). (A separate registered bug, LFV-254, can also change ContextQuery's context boost when a longer indexed context is matched through a prefix context or addAllContexts(); the ContextQuery cases cited here index every suggestion under one context, so there is no longer context for that bug to act on.)
4. setNextMatch compares ref.ints[i] with pathPrefix.ints[i] position by position (:277) and keeps the longest run as the boost. No side is converted.

Boost 1, score 3 x 1 = 3.0; counting letters gives 2 and 6.0. The typo axcd (matched prefix axc) gets boost 1 either way, 4.0. So the typo ranks first.

The result is neither a code-point count nor a byte count. Every ASCII letter (0-127) has the same value in both units; the first non-ASCII query letter is compared with a single byte of the suggestion and differs, except by accident when the code point has the same value as that byte: Ã is U+00C3 and é starts with the byte C3, so for the query aÃc the typo aécd is credited with 2 letters. After such an accident the next query code point meets a continuation byte (80-BF), so a code point U+0080-U+00BF can extend the run: for the query aé (61 C3 A9) the typo aé©d (bytes 61 C3 A9 C2 A9) gets boost 3 and the exact aéd (matched prefix aÃ = 61 C3 83) only 2, so with equal weights the typo ranks first, [aé©d/3.0, aéd/2.0].

A matched path is scored right when it ends, or its comparison stops at an ASCII mismatch, before the first non-ASCII query letter: abé over abéd/3 and axéd/4 gives the correct [abéd/6.0, axéd/4.0] (the exact suggestion is scored on ab).

• Who. Applications that build FuzzyCompletionQuery with the 9-argument constructor and unicodeAware=true (the class is @lucene.experimental; the flag defaults to false), directly or wrapped in ContextQuery, over SuggestField or ContextSuggestField values.
• When. A matched path is scored wrong when its comparison with the query reaches a non-ASCII query letter: the boost is then capped at the ASCII letters before it, and is 0 when there is none (unless a code point accidentally equals a byte). A path that ends, or stops at an ASCII mismatch, before that letter is scored right (abé over abéd/3 and axéd/4: [abéd/6.0, axéd/4.0]). With the default settings the exact prefix of a query of 3 or more letters is scored on all letters but the last, so its boost is wrong whenever a non-ASCII letter comes before the last letter. In the examples, without the bug the exact prefix wins whenever its weight x 2 beats the typo's weight; with it the weights alone decide: the order changes when the typo is heavier, and with equal weights they tie.
• How bad. The accidental byte equality can give a typo the exact prefix's boost (aÃc: [aécd/8.0, aÃcd/6.0]) or a larger one: aé over aéd/1 and aé©d/1 ranks the typo first with equal weights, [aé©d/3.0, aéd/2.0].
• What stays right. Which suggestions match the query: the match automaton does not use the boost. Scores and order change, and so, under a top-n limit, does which matches are returned (aéc over aécd/3 and axcd/4 with n = 1: [axcd/4.0]; with the fix [aécd/6.0]). Nothing throws or logs. Default byte mode and ASCII-only text are unaffected. In the hunt's bounded-exhaustive comparison (ranked top-n lists for n = 1, 3 and 200) every mismatch of this bug's class was explained by the boost units alone.
• Scale and settings. Deterministic (20 iterations, always the same two failures). Executed on two-suggestion indexes and on the hunt's bounded indexes (39 suggestions; one, two or three segments, or index-sorted); nothing in the mechanism depends on index size (read). The repros use the default base codec with the completion postings format on the suggest field, as the upstream tests do; assertions on or off.
• Downstream (read, not tested). Elasticsearch's completion suggester exposes unicode_aware as a fuzzy option (elastic/elasticsearch#10746, #107707), and OpenSearch's feeds its fuzzy options into FuzzyCompletionQuery (opensearch-project/OpenSearch#22924).
• How likely. Rated plausible: deterministic for every affected lookup, but only a ranking change, easy to mistake for the known minimal-prefix limitation, and no prior report was found (searches of 2026-10-27).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<[aécd/6.0, axcd/4.0]> but was:<[axcd/4.0, aécd/3.0]>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:120)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.search.suggest.document.TestFuzzyCompletionQueryUnicodeAwareBoost.testOneNonAsciiLetter(TestFuzzyCompletionQueryUnicodeAwareBoost.java:57)

• Contract. Class javadoc (FuzzyCompletionQuery.java:43-44): "This query boost documents relative to how similar the indexed terms are to the provided prefix." DEFAULT_UNICODE_AWARE (:57-59): "Measure maxEdits, minFuzzyLength, transpositions and nonFuzzyPrefix parameters in Unicode code points (actual letters) instead of bytes." Constructor (:123): "@param unicodeAware treat prefix as unicode rather than bytes". isUnicodeAware (:227): "lengths are measured in unicode code-points rather than bytes".
• Intent on record. LUCENE-6459 (#7518), which added the class: "Documents are 'boosted' by the number of matching prefix letters of the suggestion with respect to the original term text" and "boost = # of prefix characters matched". Its patch author, in the comments: FuzzyCompletionWeight "would count the number of prefix characters a matched input has w.r.t. the analyzed query prefix and set the boost for it to the number of common prefix length". A Lucene committer in elastic/elasticsearch#24903 (2019): "a boost per path that is equal to the length of the shared prefix". None of them mentions unicode, code points or bytes.
• Tests. TestFuzzyCompletionQuery.testFuzzyQuery pins the boost as the shared-prefix count (suaggestion 4 x 2, suggestion 2 x 3, sugfoo 1 x 3, ssuggestion 1 x 1). No upstream test builds FuzzyCompletionQuery with unicodeAware=true.
• History. The comparison loop is unchanged since the class was added (git blame -w -M -C); no commit, CHANGES or MIGRATE entry or discussion calls the mixed units a choice.
• Controls. On the same Cyrillic data the default byte mode ranks the exact prefix first ([коты/15.0, киты/12.0]), and the Latin twin does too; only the documented unicode option breaks it.

The strongest case for "works as designed": the class is @lucene.experimental; no document gives the boost as a number or names its unit; the unicodeAware javadoc lists only the edit parameters; CompletionWeight.setNextMatch documents pathPrefix only as "the prefix of a matched path"; and fuzzy completion scoring already has an accepted weak spot, the minimal-prefix NOTE in setNextMatch, which a committer defended in elastic/elasticsearch#24903 ("I agree that it can be misleading but this limitation is needed"), with the same symptom: exact matches tie with typos.

Why it fails: #24903 is ASCII and comes from which path is scored; it cannot cut the count at the first non-ASCII letter, give 0 to an all-Cyrillic query, or match a code point with an unrelated byte. The Latin control and byte mode on the same Cyrillic data pass, and fixing either problem leaves the other. Cross-model reviews (OpenAI GPT-5.6) rated the intended-behaviour argument moderate and still concluded real; the devil's advocate and the adjudication found no argument that survives. Verdict: a defect.

The recommended fix was applied in a scratch worktree at a9779ef3c193 and run through gradle (seed DEADBEEF), with assertions on and off; the worktree was removed afterwards. The hunt's first version of the same idea was validated earlier by shadowing the fixed class into the stock jars. Options B and C were not run.

Give FuzzyCompletionWeight the unicodeAware flag (from createWeight) and, at the top of setNextMatch, turn the UTF-8 byte labels into code points with Util.toBytesRef and IntsRefBuilder.copyUTF8Bytes into two reused buffers; the comparison loop stays as it is. Units then match the javadoc's "code points (actual letters)". Byte mode is untouched. The accepted path is valid UTF-8 by construction, because the match automaton is the UTF32ToUTF8 conversion, so decoding is well defined.

@Override
protected void setNextMatch(IntsRef pathPrefix) {
  if (unicodeAware) {
    // refs hold code points, but the matched path holds UTF-8 byte labels: compare letters
    codePointScratch.copyUTF8Bytes(Util.toBytesRef(pathPrefix, utf8Scratch));
    pathPrefix = codePointScratch.get();
  }
  // ... the existing comparison loop, unchanged
}

Trade-offs

• One decode per matched prefix path. The byte and int buffers are reused once grown, but IntsRefBuilder.copyUTF8Bytes calls UnicodeUtil.UTF8toUTF32, which creates one small UTF8CodePoint object per non-empty decode (the JIT may remove it); performance was not measured.
• The weight already keeps per-match state (currentBoost), so the buffers add no new thread-safety concern.
• Behaviour change: scores rise for unicode-aware users with non-ASCII text; say so in a CHANGES entry (Bug Fixes) and add the repro as a test (no upstream test covers unicodeAware=true today).

What the checks validated

• Scratch worktree at a9779ef3c193, gradle, seed DEADBEEF: this doc's repro passes 3 of 3 with assertions on and off, and checkGoogleJavaFormat passes on it; with the finder's test, TestFuzzyCompletionQuery and TestContextQuery, 24 of 24 pass, on and off.
• The whole suggest module with both repro tests added: 33 classes, 242 tests, 7 skipped, 0 failures; ecjLintMain, ecjLintTest, forbiddenApisMain, forbiddenApisTest and checkGoogleJavaFormat pass.
• Without the fix the same worktree fails both repro tests 2 of 3, with assertions on and off.
• The fact check shadowed the same fix into the stock jars: the reproducer's test passes 12 of 12 and the devil's advocate's 6 of 6 (ContextQuery, 2-, 3- and 4-byte letters, aÃc); aé then ranks [aéd/2.0, aé©d/1.0] and the top-1 aéc lookup returns [aécd/6.0].

Decode the path with new String(bytes, UTF_8).codePoints().toArray() on every match.

Trade-offs

Allocates a byte array, a String and an int array per matched path; the reused-buffer version avoids that.

What the checks validated

With the fixed class shadowed into the stock jars: the finder's test 3 of 3, the independent reproducer's 12 of 12 (the adjudication's run), the intended-lens reviewer's and the finder's tests together 6 of 6, the refute reviewer's test and the 3 upstream TestFuzzyCompletionQuery tests; in the hunt's bounded-exhaustive comparison, re-run with it and the hunt's other candidate fix shadowed, the unit-mismatch class is gone and only unrelated classes remain; back-ported to 5.3.0 it turns the probe from fail to pass. The reproducer's own variant (a hand-written decoder) also passes the upstream TestFuzzyCompletionQuery, TestContextQuery, TestPrefixCompletionQuery and TestSuggestField (47 of 47).

In createWeight, keep the refs as UTF-8 bytes; in setNextMatch compare bytes but add 1 to the count only when a whole code point has matched.

Trade-offs

No per-match decode; the same results as the recommended option in principle, with more logic in the loop. Not run.

Encode the refs to UTF-8 and count matching bytes.

Trade-offs

Consistent, but the boost becomes a byte count: a shared first byte earns partial credit and a non-Latin letter counts 2-4 times. By hand, коты and киты would tie at 12. Not run.

Leave unicodeAware at its default (false). Byte mode ranks the exact prefix first on the Cyrillic example ([коты/15.0, киты/12.0]).

Trade-offs

Then maxEdits, nonFuzzyPrefix and minFuzzyLength count UTF-8 bytes, so a typo on a multi-byte letter can cost more than one edit (read, not tested): the problem unicodeAware exists to solve.

Method: bounded-exhaustive translation validation on the stock jar (semi-formal). No model checker, solver or prover was involved.

• A completion-suggester hunt wrote an independent reference of the document suggester from the javadoc, FuzzyCompletionWeight and the upstream TestFuzzyCompletionQuery expectations before running it, and compared it with the stock main jar: every suggestion of length 1-3 on three alphabets ({a, b, c}, {a, é, b}, {a, U+65E5, U+1F600}), 4 index layouts, SuggestField and ContextSuggestField, every key of length 0-3 and 240 FuzzyCompletionQuery settings (maxEdits 0-2, transpositions, nonFuzzyPrefix 0-3, minFuzzyLength 0-4, unicodeAware) plus prefix and regex queries: 2,085,120 checks per run, with assertions on and off.
• This bug's mismatch class (68,832 plain and 282,264 ContextQuery checks, plus 25,816 that also carry a separate ContextQuery bug) appeared only with unicodeAware=true on the non-ASCII alphabets; the hypothesis "query code points compared with UTF-8 bytes" explained every case, 0 unexplained. With this fix (and the hunt's other candidate fix) shadowed, the class disappears. Vacuity: a deliberately wrong reference ("fuzzy boost + 1") produced 857,592 unexplained mismatches, so the comparison does see boost errors. The logs also hold other classes from other code paths: a separate ContextQuery context-boost bug (LFV-254), regex-query findings and, with assertions on, assertion errors in ContextQuery.setInnerWeight.
• The finder then wrote a public-API JUnit (2026-10-27), checked by an independent reproducer, two executed reviews by a model from another family (OpenAI GPT-5.6), an intended-behaviour devil's advocate, an adjudication, a history sweep, a prior-report search, a security screen and a credibility audit.