FSTCompletionLookup ignores exactMatchFirst=false for by-weight lookups after build() and load(), so an exact match is always ranked first

FSTCompletionLookup has a constructor argument exactMatchFirst; its javadoc says that with false exact matches "appear in the order of discretized weight and alphabetical within the bucket". The lookup keeps two FSTCompletion views over one FST: a by-weight view for lookup(key, true, num) and an alphabetical view for lookup(key, false, num). build() and load() pass the flag only to the alphabetical view; the by-weight view is created with the one-argument FSTCompletion(FST), "higher weights first and exact first". So by-weight lookups always put the exact match first, whatever its weight, with no error, and on distinct keys the argument has no visible effect at all. With star (1), star wars (100), start (50) and 10 buckets, lookup("star", true, 3) returns [star/0, star wars/6, start/3] instead of [star wars/6, start/3, star/0]. It came with the class in SOLR-2888 (3.6.0), which replaced FSTLookup, a class that honoured the flag; every release since is affected. Two statements fix it: create the by-weight view with new FSTCompletion(fst, true, exactMatchFirst) in build() and load(), as the pre-built-completion constructor already does.

• Buckets: build() sorts the entries by weight and splits them into buckets equal ranges, bucket = (int) (line * buckets / inputLineCount) (FSTCompletionLookup.java:222-227). A result shows the bucket, not the weight: star wars/6. With 3 entries and 10 buckets: star (weight 1) gets 0, start (50) gets 3, star wars (100) gets 6.
• Two views: higherWeightsCompletion (by weight, highest bucket first) and normalCompletion (alphabetical), two FSTCompletion wrappers over the same FST (FSTCompletionLookup.java:96-100). lookup(key, contexts, higherWeightsFirst, num) picks the first when higherWeightsFirst is true (:268-272); Lookup.lookup(key, onlyMorePopular, num) delegates to it (Lookup.java:193-195).
• Exact first: an FSTCompletion with exactFirst set looks for the key itself, puts it at the head of the results and removes it from the rest (FSTCompletion.java:233-283).

1. The buckets constructor FSTCompletionLookup(Directory, String, int, boolean) stores exactMatchFirst (FSTCompletionLookup.java:130-136).
2. build() ends with this.higherWeightsCompletion = builder.build(); (:242) and this.normalCompletion = new FSTCompletion(higherWeightsCompletion.getFST(), false, exactMatchFirst); (:243-244). FSTCompletionBuilder.build() returns new FSTCompletion(automaton) (FSTCompletionBuilder.java:197), and that constructor is "Defaults to higher weights first and exact first", this(automaton, true, true) (FSTCompletion.java:124-131).
3. load() does the same with new FSTCompletion(new FST<>(readMetadata(...), input)) (FSTCompletionLookup.java:305-306); again only normalCompletion gets the flag (:307-308).
4. In the alphabetical view the flag cannot show: when more than one bucket is in use the results are sorted before they are cut (FSTCompletion.java:202-206), and an exact match is a prefix of all its completions, so it sorts first anyway; with one bucket the depth-first walk already emits it first. On stock main, instances built with true and with false gave 0 different results over about 300,000 lookups in each mode on random dictionaries of distinct keys; with the fix about 37,000 by-weight lookups differ.
5. The fourth constructor, FSTCompletionLookup(Directory, String, FSTCompletion, boolean), creates both views with the flag (FSTCompletionLookup.java:152-153) and returns the documented order.

star (1), star wars (100), start (50), 10 buckets, exactMatchFirst=false: lookup("star", true, 3) = [star/0, star wars/6, start/3], expected [star wars/6, start/3, star/0]; lookup("star", true, 1) = [star/0], expected [star wars/6]. The same after store() + load(). The alphabetical lookup("star", false, 3) = [star/0, star wars/6, start/3], which is correct UTF-8 order. Another dictionary, {foo: 1, bar: 5, fool: 200, food: 500, foobar: 1000}: lookup("foo", true, 1) = [foo/0] instead of [foobar/8]. The smallest case: {a: 0, aa: 1}, lookup("a", true, 1) = [a/0] instead of [aa/5].

• Who. Code that calls the buckets constructor (Directory, String, int, boolean) with exactMatchFirst=false, for example to rank suggestions purely by popularity, and then lookup(key, true, num), after build() or after load().
• When. For distinct keys and num > 0: whenever the typed text is itself an entry and another of its completions is in a higher bucket: ordinary autocomplete input, such as "star" while "star wars" is far more popular. Deterministic; no scale, concurrency or unusual data needed.
• What goes wrong. Silent ranking error. If num covers all completions only the order is wrong; if num is small the exact match takes a slot and the lowest-ranked correct entry drops out, so the top-1 suggestion can be the least popular entry.
• Not affected. The default (FSTCompletionLookup(dir, prefix) uses true, FSTCompletionLookup.java:116-118), alphabetical lookups, get(key) (it reads normalCompletion, :287-290), and the pre-built-completion constructor. The class is @lucene.experimental (:71).
• Size of the check. A bounded-exhaustive differential on the stock jars over every set of 1-3 distinct keys from the 39 strings of length 1-3 over {a, b, c}, with weights {0, 1, 2, 2^31-1} (596,908 dictionaries, about 4.76 billion lookups) found 265,860 order mismatches each for 3 and 10 buckets with false, and none with true or with 1 bucket. That full scope was not re-run with the fix; a smaller exhaustive run with the fix shadowed (keys of length 1-2, sets of 1-4 keys: 141,904 dictionaries, about 1.13 billion lookups) found no FSTCompletionLookup mismatch.
• Distinct keys only. Every check used dictionaries of distinct keys; a dictionary holding the same key twice, with weights in different buckets, was not covered, so the doc makes no claim about it.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: expected:<[star[ wars/5, star/0]]> but was:<[star[/0, star wars/5]]>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.search.suggest.fst.TestFSTCompletionLookupExactMatchFirstFalse.testExactMatchNotPromotedAfterBuild(TestFSTCompletionLookupExactMatchFirstFalse.java:32)

• The contract. FSTCompletionLookup.java:126-128: "@param exactMatchFirst If <code>true</code> exact matches are promoted to the top of the suggestions list. Otherwise they appear in the order of discretized weight and alphabetical within the bucket." "Order of discretized weight" only describes the by-weight path, the one that ignores the flag. The same text is on the pre-built-completion constructor (:142-144), which honours it (:152-153).
• The lower-level switch is documented and tested. FSTCompletion's exactFirst: "Find and push an exact match to the first position of the result list if found" (FSTCompletion.java:109-110). TestFSTCompletion.testRequestedCount asserts that new FSTCompletion(fst, true, false) does not promote the low-weight "one" (TestFSTCompletion.java:136-139). The sibling WFSTCompletionLookup honours exactFirst=false (TestWFSTCompletion.testNonExactFirst). No upstream test builds FSTCompletionLookup with false.
• History. SOLR-2378 added exact-match promotion "(weighted mode)" to the old FSTLookup, and SOLR-2762 fixed it there, so the flag was a feature of the weighted path. SOLR-2888 replaced FSTLookup with FSTCompletionLookup and switched Solr's FSTLookupFactory from new FSTLookup(buckets, exactMatchFirst) to new FSTCompletionLookup(buckets, exactMatchFirst) as a drop-in; its CHANGES entry announces no change in behaviour. Executed on the same input: 3.5.0 FSTLookup(10, false) returns [star wars, start, star]; 3.6.0 FSTCompletionLookup(10, false) returns [star/0, star wars/6, start/3]. Three days after the class was committed, its author added the "Otherwise ..." javadoc (3408defc8db, "JavaDoc clarifications").

The strongest works-as-designed case (built by the intended-behaviour verifier, reviews by a model from another family, OpenAI GPT-5.6, and the adjudication): FSTCompletionBuilder's class javadoc says "If there is an exact match in the automaton, it is returned first on the results list (even with by-weight sorting)" (FSTCompletionBuilder.java:77-78), its no-argument constructor makes an FSTCompletion "with default options: 10 buckets, exact match promoted to first position" (:127-131), and FSTCompletion(FST) defaults to exact first; the class is @lucene.experimental; nobody complained in about 15 years; and for load(), the no-argument constructor "should only be used to read a previously saved suggester" (FSTCompletionLookup.java:105), defaults to true, and store() does not record the flag. Nearby javadoc is also loose: FSTCompletion says higherWeightsFirst=false "has no effect" (FSTCompletion.java:106-108), yet lookup then sorts alphabetically when more than one bucket is in use (:202-206), and Lookup documents the by-weight switch as onlyMorePopular, "return only more popular results" (Lookup.java:189).

Why it fails: the builder and one-argument docs describe the default view, which the explicit argument exists to override, and its own javadoc says so; experimental status allows API changes, not a documented argument that does nothing; few users set false, and Solr's modern suggest component uses the unaffected alphabetical path; load() itself applies this.exactMatchFirst to normalCompletion, nothing forbids loading into a configured instance, Solr does exactly that, and the build() path alone proves the bug. The exactMatchFirst text, unlike the loose sentences, is specific to this class and argument, was added by the class's author three days after the class, and is implemented by the pre-built-completion constructor. Every check rated the intended argument weak; the cross-model fact check of this doc rated it weak for build() and moderate for load().

One fix was applied in a scratch worktree at a9779ef3c193 and run through gradle (seed DEADBEEF), together with a regression test in TestFSTCompletion; the alternatives were not run.

Two statements in FSTCompletionLookup.java (+4/-2 lines after wrapping): create higherWeightsCompletion as new FSTCompletion(fst, true, exactMatchFirst), as the pre-built-completion constructor already does (:153). The FST and the stored format do not change.

       // The two FSTCompletions share the same automaton.
-      this.higherWeightsCompletion = builder.build();
+      this.higherWeightsCompletion =
+          new FSTCompletion(builder.build().getFST(), true, exactMatchFirst);
 ...
     this.higherWeightsCompletion =
-        new FSTCompletion(new FST<>(readMetadata(input, NoOutputs.getSingleton()), input));
+        new FSTCompletion(
+            new FST<>(readMetadata(input, NoOutputs.getSingleton()), input), true, exactMatchFirst);

Trade-offs

• The default exactMatchFirst=true behaves exactly as before.
• Behaviour change for users who set false: by-weight lookups now follow the documented order; worth a CHANGES line. Anyone who relied on the old order can pass true, which gives it for distinct keys (duplicate keys were not checked).
• builder.build() still makes one throwaway FSTCompletion (it caches at most one root arc per bucket), negligible next to building the FST.

What the checks validated

• Stock main a9779ef3c193 (gradle, seed DEADBEEF): the repro fails 2 of 3 and the new TestFSTCompletion#testExactMatchFirstFalse fails, with assertions on and off; the control and the 13 upstream TestFSTCompletion tests that run pass (testRandom is @Nightly).
• With the fix: the repro and TestFSTCompletion pass (18 tests, 0 failures, 1 @Nightly skipped), with assertions on and with assertions off.
• With the fix, the whole :lucene:suggest:test: 32 classes, 240 tests, 0 failures, 7 skipped (31 upstream classes, 236 upstream tests, plus the repro and the new test); this run had no -Ptests.asserts, so it ran without assertions, the default for seed DEADBEEF.
• ecjLint and checkGoogleJavaFormat pass for lucene/suggest with both tests in place; both diffs apply cleanly to a clean tree.
• Through the classpath runner the fix shadowed onto the stock jars passes the repro 3 of 3; in the hunt, the finder's, the independent reproducer's, the cross-model reviewers' and the adjudication's tests passed with this change or with an independently written equivalent of it.
• Not run: other modules (nothing outside lucene/suggest uses the class); no benchmark (the change only passes a flag).

"If there is an exact match in the automaton, it is returned first on the results list (even with by-weight sorting)" (FSTCompletionBuilder.java:77-78) describes the default FSTCompletion that build() returns; adding "by default" removes the only text that reads as the opposite contract.

Trade-offs

Doc-only.

fix-with-test.diff adds TestFSTCompletion#testExactMatchFirstFalse: build and store/load with false, assert [star wars/6, start/3, star/0]. It fails on stock main and passes with the fix.

Trade-offs

None; no upstream test builds FSTCompletionLookup with false today.

Change the javadoc to say that false only affects alphabetical lookups.

Trade-offs

Makes the argument a no-op on distinct keys and keeps the regression from FSTLookup. Note also open #11576, which proposes removing the alphabetical mode: without the fix, the flag would then reach no view at all.

Method: bounded-exhaustive differential translation validation on the stock jars (semi-formal tier), in a hunt unit that compared the suggest module's Lookup implementations (all except FreeTextSuggester).

• The harness builds every implementation from the same dictionary and compares each lookup with an independent reference written from that class's javadoc; for FSTCompletionLookup: bucket descending, then UTF-8 order, the exact match first only when exactMatchFirst is true.
• Scope: every set of 1-3 distinct keys over the 39 strings of length 1-3 over {a, b, c}, with weights {0, 1, 2, 2^31-1}: 596,908 dictionaries, 7,759,804 builds and 4,763,325,840 lookups, assertions off. The class "exact=false, order" appeared 265,860 times for 3 buckets and as often for 10, never for true or 1 bucket. With the fix shadowed onto the jar (141,904 dictionaries including 4-key sets, about 1.13 billion lookups) it appeared 0 times.
• The class was reduced to a public-API JUnit (found 2026-10-27 on main ea57360912ed, whose lucene/suggest equals a9779ef3c193), then checked by an independent reproducer, an intended-behaviour devil's advocate, two executed reviews by a model from another family (OpenAI GPT-5.6), an adjudication, a history and prior-report follow-up and a credibility audit.