AnalyzingSuggester counts skipped duplicate copies toward maxSurfaceFormsPerAnalyzedForm, so a lower-ranked variant silently disappears

AnalyzingSuggester and FuzzySuggester keep at most maxSurfaceFormsPerAnalyzedForm surface forms (default 256) for texts that the analyzer maps to the same analyzed form, and they remove repeated copies of a surface form, keeping the highest-weighted one. build() applies the limit before it checks for a repeat (AnalyzingSuggester.java:544-554), so every skipped copy still uses up a slot. With StandardAnalyzer, the default constructor and a DocumentDictionary over 256 documents Apple iPhone (weights 100 to 355) plus one apple iphone (weight 5), lookup("apple ip", false, 2) returns only [Apple iPhone/355], and the exact text apple iphone is not returned either; 255 documents return both. With the limit set to 2, a/10, a/5, A/3 gives [a/10]. Nothing throws. 4.1.0 to 10.5.2 and main are affected (10 releases and main run; the same code order in every release tag of that range that was read), and checking for the repeat before counting fixes it.

• Surface form: the suggestion text as given, for example Apple iPhone. Analyzed form: what the analyzer makes of it, here apple iphone for both Apple iPhone and apple iphone. Lookups match on the analyzed form and return surface forms.
• maxSurfaceFormsPerAnalyzedForm (constructor argument, default 256, at most 256): how many surface forms are kept per analyzed form; the javadoc says the lowest-weighted ones are discarded beyond that.
• Repeated copies: the same surface form given more than once, for example by a DocumentDictionary, which yields one entry per stored value of the field in each live document (one per document for a single-valued field). build() keeps the highest-weighted copy and skips the rest.

1. build() sorts all entries by analyzed form, then by weight (highest first), then by surface bytes (AnalyzingComparator, AnalyzingSuggester.java:337-385).
2. It walks them with a counter dedup per analyzed form. The counter is also a one-byte suffix that keeps each kept form's FST key unique (:570-571), which is why the limit is capped at 256 (:229-238).
3. For each further entry of the same analyzed form it runs dedup++ (:545), then cuts the entry if dedup >= maxSurfaceFormsPerAnalyzedForm (:546-550), and only then skips it if the surface form was already kept (:551-553). A skipped copy is never added to the FST, but it has already raised the counter.
4. A new analyzed form resets the counter and the seen-set (:555-559). FuzzySuggester extends AnalyzingSuggester and does not override build() (FuzzySuggester.java:66).

lookup("a", false, 5) returns [a/10] instead of [a/10, A/3].

256 documents Apple iPhone (weights 100 to 355) and one apple iphone (weight 5), StandardAnalyzer, default constructor, DocumentDictionary: the 256 copies take counter values 0 to 255, apple iphone gets 256 and is cut. lookup("apple ip", false, 2) returns [Apple iPhone/355], and lookup("apple iphone", false, 2) returns [Apple iPhone/355] instead of [apple iphone/5, Apple iPhone/355] (the exact-match-first pass, :716-781, cannot find a form that is not in the FST). With 255 documents the variant gets counter 255 and both are returned. FuzzySuggester gives the same results.

• Who. Users of AnalyzingSuggester and FuzzySuggester (both @lucene.experimental). DocumentDictionary yields one entry per stored value of the field in each live document, so a field whose values repeat (product names, brands, categories, tags, popular titles) reaches 256 copies easily; a FileDictionary with repeated lines does too.
• What else it takes. A distinct variant of the same analyzed form that ranks below the copies (a lower weight, or the same weight and later in byte order, AnalyzingSuggester.java:369-384; the equal-weight case was read, not executed): in practice a case, accent or punctuation variant under a lowercasing or folding analyzer. By reading (not executed), also texts that a stop-word or synonym analyzer maps to the same bytes.
• Default settings. The 3-argument constructors (limit 256, EXACT_FIRST and PRESERVE_SEP) and StandardAnalyzer are enough. With a lowered limit k, k copies are enough.
• Silent. No exception and no log; the suggester is valid, just smaller. The lookup returns fewer results than asked for (or, by reading, the next suggestion moves up into the variant's place), and a user who types the variant exactly does not see it.
• The threshold is exact. A distinct form is dropped once at least limit entries of its analyzed form, copies included, sort ahead of it: 255 copies keep it, 256 drop it. Distinct forms that rank above the copies, and other analyzed forms, are not affected.
• Downstream (read, not tested): Solr's FuzzyLookupFactory and AnalyzingLookupFactory build these suggesters with this limit (default 256, configurable). The Elasticsearch completion suggester uses the separate suggest.document package, which this bug does not touch; for OpenSearch the search found no production use of these two classes, but it was not exhaustive.
• How likely. Rated rare: the variant must share the analyzed form and rank below 256 copies, the main spelling is still suggested, so the loss is easy to miss, and nothing has been reported since 2013.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: expected:<[a/10[, A/3]]> but was:<[a/10[]]>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.search.suggest.analyzing.TestAnalyzingSuggesterDuplicateCopyTakesSlot.testDuplicateCopyDoesNotUseSurfaceFormSlot(TestAnalyzingSuggesterDuplicateCopyTakesSlot.java:36)

• Contract. Constructor javadoc (AnalyzingSuggester.java:205-206, the same in FuzzySuggester.java:138-139): "Maximum number of surface forms to keep for a single analyzed form. When there are too many surface forms we discard the lowest weighted ones." Field javadoc (:141): "Maximum number of dup surface forms (different surface forms for the same analyzed form)." Comment above the seen-set (:511-514): "Used to remove duplicate surface forms (but we still index the highest-weight one)." The repro has 2 distinct surface forms.
• Intent. #5599 (LUCENE-4533, 2012, "if you pass dups (same surface form) to AnalyzingSuggester it can fail to return enough results"), by @mikemccand: "I think we should de-dup the user's input (keeping highest weight for each surface form) before adding to the FST." Their fix comment bounds the seen-set ("worst case is it holds 256 keys") but does not discuss how copies count toward the limit.
• History. The counter and the cut date from LUCENE-3842 (f2f91bae46c, 2012-09-28), when every counted entry was indexed with its own suffix. The LUCENE-4533 fix (37cdda05f50, 2012-11-05) put the seen-set check below the existing cut.
• Tests. No test asserts the current behaviour. testMaxSurfaceFormsPerAnalyzedForm uses 3 distinct forms and no copies; testDupSurfaceFormsMissingResults3 feeds repeated entries on purpose, with 5 entries; testRandom avoids repeated surface forms. A weaker analogy: the sibling document suggester's testExtremeDeduplication indexes at least 500 copies of abc plus one abd and expects both, so copies are not meant to crowd out a different suggestion there; but it is different code, and abc and abd are different analyzed forms, so it does not test a per-form cap.
• Legal input. Nothing in InputIterator, Lookup.build, DocumentDictionary, FileDictionary, CHANGES or MIGRATE requires unique input.
• The exact-match comment. A comment in lookup (:737-741) says the cap may have "pruned our exact match": the authors accepted that the cap can remove a form someone types exactly. That fits a cap on distinct forms and does not say identical copies count; here the cut happens with only 2 distinct forms, so the exact-match loss is a consequence of the wrong cut, not a separate defect.

The strongest case for "works as designed" (the cross-model reviews, the reproducers and the adjudication looked for it): the limit is also an implementation bound, the one-byte suffix ("this is just an implementation limitation"), and the cut's comment reads "More than maxSurfaceFormsPerAnalyzedForm dups: skip the rest", so the counter could be read as a cap on raw entries; the classes are @lucene.experimental; callers could be expected to pre-aggregate (strongest for a FileDictionary or a custom iterator, weakest for a DocumentDictionary, whose entries come from the index); the authors knew the cap can prune an exact match (:737-741); today's order also skips the seen-set lookup for every entry after the cut; and the lost forms are low-ranked variants.

Why it fails: skipped copies never use a suffix byte, so counting them protects nothing the bound exists for, and the fix keeps the suffixes below the limit and the seen-set bounded by it; the saved lookups are not a documented contract; the javadoc defines the capped items as different surface forms; #5599 asked for dedup before the FST; the exact-match comment fits a cap on distinct forms, and here the cut happens with only 2 of them; and the outcome flips at an arbitrary copy count (255 kept, 256 lost). The cross-model reviews and the independent reproduction rated the intended-behaviour argument weak explicitly, and the intended-behaviour check and the adjudication rejected it; a later cross-model fact check of this doc rated it moderate for exact repeats from a FileDictionary or custom iterator and weak for DocumentDictionary. Verdict: a defect.

Option A was built and checked in a scratch worktree; B, a smaller variant, was checked on the repro, the fix check and the analyzing tests; D is a workaround for released versions.

The finder's change, +6/-5 in AnalyzingSuggester.build. A copy of a kept surface form is skipped first and does not count; a new distinct form is cut if it would exceed the limit, otherwise it takes the next counter value. Its comment should say "lower- or equal-weighted", since equal-weight copies are skipped too.

} else if (analyzed.get().equals(previousAnalyzed.get())) {
  if (seenSurfaceForms.contains(surface)) {
    // a lower weighted copy of a surface form we already kept: it does not count
    continue;
  }
  if (dedup + 1 >= maxSurfaceFormsPerAnalyzedForm) {
    // More than maxSurfaceFormsPerAnalyzedForm
    // dups: skip the rest:
    continue;
  }
  dedup++;
  seenSurfaceForms.add(BytesRef.deepCopyOf(surface));

Trade-offs

• Behaviour change on rebuild: repeated input now keeps up to limit distinct forms per analyzed form, as documented; the FST grows by at most limit - 1 entries per analyzed form, the same bound as for unique input. Input whose analysis yields no repeated (analyzed form, surface form) record builds exactly as before. Distinct input alone does not guarantee that: a graph analyzer can yield repeated analyzed paths (build does not determinize, and FiniteStringsIterator may return duplicates for a non-determinized automaton), and fix A stops counting those too (read, not executed).
• No format change: store/load are untouched; kept forms get contiguous suffix bytes 0 to k-1 instead of gaps. Stored suggesters load as before and keep their old content until rebuilt.
• Bounds kept: the seen-set holds only kept forms (at most limit) and suffixes stay below the limit.
• Cost: the seen-set lookup now runs for every entry of an analyzed form, also past the limit, where stock code skips it (the cut at :546-549 comes first): more hash lookups for analyzed forms with more than limit entries. The set stays bounded; the offline sort has already handled every entry. Not benchmarked.
• Payloads as today: the sort and the seen-set compare surface bytes, not payloads (:369-384, :551), so when equal-weight copies of one surface form carry different payloads, which payload is kept depends on the sorted order. Fix A does not change that.

What the checks validated

• The repro: 4 of 5 fail on stock, 5 of 5 pass with the fix (gradle with assertions on and off; the repro runner with -ea and -da).
• The fix check: 2 of 2 fail on stock, 2 of 2 pass (-ea and -da).
• :lucene:suggest:test --tests 'org.apache.lucene.search.suggest.analyzing.*': 96 tests (89 upstream + 7 new), 0 failed, 1 skipped, with assertions on (seed 5EED7248) and off (seed 77AA2480).
• All of :lucene:suggest:test: 243 tests (236 upstream + 7 new), 0 failed, 7 skipped (6 @Ignore, 1 @Nightly, so 229 upstream tests ran), in both modes, repeated by the fact check at a fresh seed; :lucene:suggest:check -x test (format, ecjLint, forbiddenApis) passes.
• Earlier checks: the finder's and the reproducers' tests pass with it, also in a separate re-run; the bounded-exhaustive harness finds 0 mismatches in its 3-entry lowercase scope; the 4.1.0 source with the fix passes the finder's repro.

Swap the two if blocks (+3/-3). The repro passes 5 of 5 and the fix check 2 of 2 (-ea, and -da in the fact check), and the 96 analyzing tests with assertions on; format passes.

} else if (analyzed.get().equals(previousAnalyzed.get())) {
  if (seenSurfaceForms.contains(surface)) {
    continue;
  }
  dedup++;
  if (dedup >= maxSurfaceFormsPerAnalyzedForm) {
    continue;
  }
  seenSurfaceForms.add(BytesRef.deepCopyOf(surface));

Trade-offs

Past the limit the counter keeps growing for every entry that is not a copy of a kept form (further distinct forms, and copies of cut forms, which never enter the seen-set; copies of kept forms are still skipped without counting), so it no longer equals the number of kept forms; by reading, it could misbehave only after about 2^31 entries of one analyzed form (int overflow). A keeps the counter exact. The whole suggest module was not run with B.

Change the javadoc instead of the code.

Trade-offs

Keeps today's output but contradicts the field javadoc and the intent of #5599, and leaves the 255/256 cliff that depends on how often another text repeats.

Wrap the InputIterator so each text is passed once with its highest weight, which is what build() keeps anyway. Validated on main, 10.5.2 and 9.12.3.

Trade-offs

Memory proportional to the number of distinct texts. The sketch drops payloads and contexts; a real wrapper must keep the payload of the highest-weighted copy. Alternatively aggregate at indexing time, one document per suggestion.

Method: bounded-exhaustive translation validation (the hunt's "semi-formal" tier), a hunt unit (suggesters), 2026-10-26, main ea57360912ed.

• The finder wrote a reference from the javadoc: per analyzed form, keep the maxSurfaceFormsPerAnalyzedForm highest-weighted distinct surface forms; repeated copies keep their highest weight. The hypothesis came from reading build() while writing it, and a probe confirmed it.
• On separate build machines the harness compared the real AnalyzingSuggester and FuzzySuggester with the reference for every dictionary of at most 3 entries over 12 strings and weights {0, 1, 2}, every key of up to 3 characters and num 1 to 7: 330,258 mismatches in 161,239,680 lookups (lowercasing analyzer, 12 AnalyzingSuggester configurations), 29,184 with a plain analyzer, and 409,488 in 644,958,720 (plain analyzer, 48 FuzzySuggester configurations). All were of this one class and none unexplained; there were none with 2 or fewer entries.
• 20,000 random dictionaries of up to 6 entries: 1,746,904 (lowercase) and 469,544 (plain) mismatches, all of this class. With the fix shadowed: 0 mismatches in the 3-entry lowercasing AnalyzingSuggester scope; the FuzzySuggester scope was not re-run with the fix. A seeded mutant of the reference (a flipped tie-break) was caught (31,752 unexplained mismatches), so the comparison is not vacuous.
• Exhaustive only within these bounds. The failing case was then written as a public-API JUnit.