AnalyzingSuggester and FuzzySuggester getCount() counts analyzed paths, not entries, when the analyzer makes a token graph

Lookup#getCount returns "the number of entries the lookup was built with". AnalyzingSuggester.build (inherited by FuzzySuggester) indexes every analyzed path of each entry, and it increments its counter in the header of that per-path loop (AnalyzingSuggester.java:419), so with an index analyzer that stacks tokens (a synonym graph, a word-delimiter graph with PRESERVE_ORIGINAL) each entry counts once per path: one entry fast with the rule fast -> {fast, quick} gives 2, the entries fast car and red car give 3, and the entry wifi wifi wifi with a stacked synonym gives 8. The same per-path increment leaves out an entry that analyzes to no token at all, such as an entry of only stop words. store() writes the number and load() reads it back. Suggestions are correct and no production code in Lucene calls getCount(), so only applications that show or check the count are affected. The count was per entry from 4.7.0, when it was added, to 5.2.1; LUCENE-6365 moved the increment while rewriting the loop, first released in 5.3.0. Every release run from 5.3.0 to 10.6.0 and main is affected, and moving newCount++ before the per-path loop fixes it.

1. One entry, many paths. For each entry build analyzes it into an automaton and iterates its finite strings, at most maxGraphExpansions of them (AnalyzingSuggester.java:415-417). The 3- and 4-argument constructors of both classes pass -1, no limit (:173-176, :182-196; FuzzySuggester.java:98-129). The class knows paths and entries differ: maxAnalyzedPathsForOneInput is the "Highest number of analyzed paths we saw for any single input surface form. For analyzers that never create graphs this will always be 1." (:153-157).
2. The increment is in the per-path loop header (:415-419; the end-of-line comments are ours):

   for (BytesRef surfaceForm; (surfaceForm = iterator.next()) != null; ) {       // once per entry
     LimitedFiniteStringsIterator finiteStrings =
         new LimitedFiniteStringsIterator(toAutomaton(surfaceForm, ts2a), maxGraphExpansions);

     for (IntsRef string; (string = finiteStrings.next()) != null; newCount++) { // once per path

1. Published, stored, reloaded. count = newCount at the end of the build (:590); store() writes it (:601), load() reads it back (:614), getCount() returns it (:851-852).
2. FuzzySuggester extends AnalyzingSuggester (FuzzySuggester.java:66) and overrides neither build nor getCount.

Entries with exactly one analyzed path are counted right. An entry that analyzes to no token (a blank entry, or only stop words) has no path, because the automaton TokenStreamToAutomaton builds reaches an accept state only through a position that a token created (TokenStreamToAutomaton.java:244-255), so the per-path increment leaves it out: 5.2.1 counted it (2 for the, fast car), 5.3.0 does not (1), and the fix counts it again (2). Every lookup checked in these runs returned the right suggestions.

• Who. Users of AnalyzingSuggester and FuzzySuggester (both @lucene.experimental) whose index analysis gives an entry several paths, as SynonymGraphFilter with a matching rule and WordDelimiterGraphFilter with PRESERVE_ORIGINAL on a token it splits do in the runs. The class javadoc itself shows a synonym setup (AnalyzingSuggester.java:78-80).
• Effect. A silently wrong number from getCount(): the inflation factor is the number of analyzed paths per entry, 2x for one matching synonym and, with independent two-way synonym alternatives, 2^k for k matching tokens in one entry (8 for wifi wifi wifi); the default constructors set no cap (maxGraphExpansions -1). Entries that analyze to no token are left out of the count. A suggester saved with store() and reloaded with load() reports the same wrong number until it is rebuilt.
• Not affected. Suggestions, weights, payloads and build work: every path is indexed whether or not it is counted. No production code in Lucene calls getCount() (store() only writes the field), and Solr's suggester code does not call it either (other downstream code was not surveyed).
• Likelihood: plausible. Deterministic on every such build, but only applications that read the count notice; the searches found no public report since 5.3.0.

lucene/suggest already depends on lucene/analysis/common, so the test drops into the module's test tree as is. The test framework randomizes settings such as locale and directory, but the suggester uses the directory only for temporary sort files (AnalyzingSuggester.java:396-400) and the count does not depend on the seed. It failed the same way with every seed tried: DEADBEEF through gradle with assertions on and off, and on a standalone runner with -ea and -da; 5EED0001 with -ea; FC431A01 (-ea) and FC431A02 (-da). The standalone runner records the main checkout as dirty, because of two untracked files outside lucene/suggest (neither is a .java source); the tracked diff is empty, and the gradle runs in a clean scratch checkout fail the same way.

Suggested regression test: this same test, for example as a method of TestAnalyzingSuggester.

Not shown by this test (same cause, same fix): FuzzySuggester, the store()/load() round trip, larger inflation with more stacked tokens. Two entries fast car, red car give 3 instead of 2 for both suggesters, before and after store()/load(), while lookup("quick") and lookup("fast") each return the one entry.

TestAnalyzingSuggesterGetCount > testSynonymDoesNotInflateCount FAILED
    java.lang.AssertionError: number of entries the lookup was built with expected:<1> but was:<2>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:10604C74227CF345]:0)
        at junit@4.13.2/org.junit.Assert.fail(Assert.java:89)
        at junit@4.13.2/org.junit.Assert.failNotEquals(Assert.java:835)
        at junit@4.13.2/org.junit.Assert.assertEquals(Assert.java:647)
        at org.apache.lucene.search.suggest.analyzing.TestAnalyzingSuggesterGetCount.testSynonymDoesNotInflateCount(TestAnalyzingSuggesterGetCount.java:49)

:lucene:suggest:test (FAILURE): 1 test, 1  failure
1 test completed, 1 failed

# the same with -Ptests.asserts=false, on the standalone runner with -ea and -da,
# and on the 10.6.0 and 10.5.2 release jars; with the recommended fix: passes

• Contract. Lookup#getCount: "Get the number of entries the lookup was built with" / "@return total number of suggester entries" (Lookup.java:171-176). The field: "Number of entries the lookup was built with" (AnalyzingSuggester.java:166). Lookup.build(Dictionary) feeds dict.getEntryIterator() (Lookup.java:145-146), so the entries are what the dictionary yields.
• The class separates paths from entries. maxAnalyzedPathsForOneInput counts "analyzed paths ... for any single input surface form" (:153-157); maxGraphExpansions "only matters if your analyzer makes lots of alternate paths (e.g. contains SynonymFilter)" (:144-147).
• History. LUCENE-5404 (#6467, "Add support to get number of entries a Suggester Lookup was built with and minor refactorings"; b5926da2d67 on trunk and fdf4595051a on branch_4x, which 4.7.0 ships) put count++ after the loop over an entry's paths, so every entry counted once, also one with no path. LUCENE-6365 (#7425, 3d8bfad40b0 and branch_5x 9a3ec6ce777, 2015-07-04, first released in 5.3.0) replaced that loop with an iterator loop and moved the increment into its header; its commit message, CHANGES entry and issue discussion say nothing about counting. LUCENE-10292 (#11328) renamed the counter to newCount and kept it there. That the move was an accident is an inference from this record; nobody upstream has said so.
• Tests. Every upstream getCount assertion for these classes uses MockAnalyzer, where paths equal entries (TestAnalyzingSuggester.java:181, 193, 1028, 1051; TestFuzzySuggester has none). The whole suggest suite passes with the fix.

Strongest case (the devil's-advocate verifier and the adjudication): "entries" means the suggester's internal records, the analyzed-form and surface-form pairs it writes into its FST, so counting paths is by design; getCount is loose across lookups anyway (inputs a, a, b give 2 in WFSTCompletionLookup, 3 in TSTLookup and FSTCompletionLookup); and the classes are experimental.

Why it fails: the count is not the record count either. With a keyword analyzer a, a, b gives 3 although the duplicate surface form is dropped from the FST, and fast car twice with the synonym gives 4. A documented "number of entries" that changes with the maxGraphExpansions tuning knob (3 or 2 for the same two entries) is implausible, and #6467 and the 4.7.0 to 5.2.1 code count entries. The per-path count also drops entries that 5.2.1 counted: an entry that analyzes to no token adds 0. Experimental status bears on severity, not on intent. Still, a maintainer who puts compatibility first may prefer to document today's count rather than change a stored statistic.

The recommended fix is the finder's one-line change, applied to 645d4fe14a20 in a scratch worktree with git apply and run through gradle. An equivalent placement and two alternatives follow. Fix A changes the stored count once a suggester is rebuilt, so the issue can ask the maintainers whether compatibility makes them prefer documenting the current count (option C).

In AnalyzingSuggester.build, move newCount++ out of the header of the loop over finiteStrings to the line before it.

        LimitedFiniteStringsIterator finiteStrings =
            new LimitedFiniteStringsIterator(toAutomaton(surfaceForm, ts2a), maxGraphExpansions);

-        for (IntsRef string; (string = finiteStrings.next()) != null; newCount++) {
+        newCount++; // one entry, however many analyzed paths it has
+        for (IntsRef string; (string = finiteStrings.next()) != null; ) {

Trade-offs

• Restores the 4.7.0 to 5.2.1 meaning: one per input entry, duplicates included, as TSTLookup and FSTCompletionLookup count. Entries with exactly one analyzed path count as today; an entry with several paths counts once instead of once per path, and an entry that analyzes to no token counts once instead of zero times, as in 5.2.1.
• No format change: store()/load() are untouched. Suggesters saved by affected versions keep their inflated count until rebuilt; worth a CHANGES line.
• Behaviour change to weigh: once rebuilt, a suggester whose entries have several paths or none reports a different number than before (smaller, or larger), so an application that compares counts across a Lucene upgrade sees a one-time step. A maintainer who puts compatibility first may prefer documenting the current count.
• No change to suggestions or build work: the loop body is untouched.

What the checks validated

• Stock + repro through gradle: fails (expected:<1> but was:<2>). + fix: the repro passes with assertions on and off.
• + fix: TestAnalyzingSuggester (23 tests), TestFuzzySuggester (14) and the repro pass, 38 of 38.
• + fix: the whole :lucene:suggest:test, 32 suites, 237 tests, 0 failures, 7 skipped by their own markers (assertions on); the first doc round got the same counts with assertions off.
• :lucene:suggest:check -x test (google-java-format, ecjLint, forbiddenApis, with the test in the tree): passes.
• In the hunt (fixed class shadowed): the finder's 2 tests, the independent reproducer's 6 tests and the cross-model stage's probe pass; the first fact check's probe gives the entry count for every case, including entries with no token.
• Not run: the full Lucene test suite, nightly tests, benchmarks.

Put newCount++ after the loop, next to the maxAnalyzedPathsForOneInput update (AnalyzingSuggester.java:484).

Trade-offs

Identical to the recommended fix for every entry, including one with no path: both add 1 once per entry. Executed by the fact check with the fixed class shadowed: the repro passes with assertions off, and every probe value, the zero-token cases included, equals the recommended fix's.

Say in the field comment and an AnalyzingSuggester#getCount javadoc that the count is of analyzed paths.

Trade-offs

Keeps today's numbers but contradicts Lookup#getCount, makes the number depend on maxGraphExpansions, and is still not the record count (duplicates are counted, entries with no token are not). It is the option for a maintainer who puts compatibility first.

Count the distinct analyzed-form and surface-form pairs written into the FST.

Trade-offs

Also changes the count for analyzers without graphs (a, a, b would give 2) and has no basis in the javadoc.

Method: non-formal (code reading plus an executed test), a hunt unit (counts and statistics across modules), 2026-10-26, main ea57360912ed.

• The finder inventoried what each suggester's getCount() counts and read it against the javadoc; AnalyzingSuggester.java:419 was the one place that counts analyzed paths rather than inputs (the inventory also noted that WFSTCompletionLookup counts distinct keys, an ambiguity it did not file).
• A JUnit on the stock jars (entries fast car and red car, WhitespaceTokenizer + SynonymGraphFilter, lookups asserted before the count) failed for both suggesters: 3 instead of 2.
• The history clone (git log -G on the counting line, git show of the four commits that touched it) and release runs from 4.6.1 to 10.5.2 dated the change to LUCENE-6365.