WordBreakSpellChecker.suggestWordCombinations returns no combinations when the terms start with SEPARATOR_TERM

WordBreakSpellChecker.suggestWordCombinations suggests joining neighbouring query words, for example sun + flower -> sunflower, when the joined word is in the index. Its public SEPARATOR_TERM, new Term("", ""), is documented only as the "Term that can be used to prohibit adjacent terms from being combined", and the loop skips a separator at any position, the first included. But every joined word is looked up as new Term(terms[0].field(), combinedTermText), in the field of the first array element, whichever words are being joined. With the separator first, that field is "", which the index does not have: docFreq 0, so the minimum-frequency check (default 1) drops every combination. On an index with one document f:sunflower, [SEPARATOR_TERM, f:sun, f:flower] returns no suggestion, while [f:sun, f:flower] returns sunflower; nothing is thrown, in any SuggestMode. It has been there since the class was added (LUCENE-3523, 4.0.0-ALPHA, 2012) and is in every release up to 10.6.0 and in main. Only direct callers that emit a leading separator are hit (Solr's adapter adds one only after the first token), the effect is missing spelling suggestions only, and the fix is one token: terms[i].field().

1. The separator is an ordinary Term with an empty field: public static final Term SEPARATOR_TERM = new Term("", "") (:41), documented as the "Term that can be used to prohibit adjacent terms from being combined" (:40).
2. The loop accepts it anywhere. For each left term i, a separator is skipped (:169-172); for each right term j, a separator ends the run (:186-189). This generic check also skips i = 0, so with a leading separator the runs start at i = 1.
3. The lookup ignores i. For each run i..j the joined text is looked up as new Term(terms[0].field(), combinedTermText) with ir.docFreq(combinedTerm) (:203-204). With terms[0] = the separator, every run is looked up in field "".
4. The repro's index has no field "", so docFreq is 0 (in field f, sunflower has docFreq 1). The check combinedTermFreq >= minSuggestionFrequency (:208, default 1 at :34) drops the combination. The SUGGEST_MORE_POPULAR and SUGGEST_WHEN_NOT_IN_INDEX conditions (:206-207) only add requirements, so with default settings every mode returns nothing. With setMinSuggestionFrequency(0) the combination comes back with freq 0 instead of 1. Empty field names are legal, so in an index that has a field "" the frequency, and whether the word is offered, come from that unrelated field.
5. Every other lookup uses the term's own field: the original frequencies ir.docFreq(terms[i]) (:159) and the suggestWordBreaks path (term.field() in generateBreakUpSuggestions, :278, :281, :312, :315). :203 is the only place that takes the field from position 0.

Side note, not claimed: the same line also decides the field when one call passes terms of several fields: in [f:new, f:york, SEPARATOR_TERM, g:sun, g:flower] the g combination sunflower is counted in field f. No documented contract covers mixed-field calls.

• Who is exposed. Callers of WordBreakSpellChecker.suggestWordCombinations (module lucene/suggest) whose term array starts with SEPARATOR_TERM. A natural way to get there: add a separator whenever the clause type (required, prohibited, plain) changes, with the "previous clause" flags starting at "plain"; for +sun +flower the change fires before the first term.
• Solr's adapter does not build this input. WordBreakSolrSpellChecker (apache/solr main, the one adapter inspected) adds the separator only if (i > 0 && ...) and builds every term in its one configured field. Other integrations and custom adapters were not inspected.
• Default vs opt-in. Default settings, no special analyzer or codec. One document is enough; the index only needs the joined word in the terms' field. The input shape is uncommon, and no upstream test uses SEPARATOR_TERM at all.
• What breaks. That call returns no word-combination suggestions, silently. Search results are not affected; nothing is thrown, written or corrupted.
• How likely. Rare: it needs a direct-API caller that emits a leading separator, and no public report was found. Workaround: drop a leading separator before the call (or add separators only for i > 0).

The test is the hunt's minimal repro with its comment line shortened to fit 100 columns; it passes :lucene:suggest:check -x test. It runs unchanged on 10.6.0 and 10.5.2; on 9.x write suggestions[0].suggestion.string (a field there) instead of suggestion().string.

Executed: gradle on main 645d4fe14a20 with assertions on (seed 5EED1073) and off (5EED1074), both fail; the standalone runner on the 10.6.0 release jars with assertions on and off, both fail; the hunt's runs on main, 10.5.2 and 9.12.3 fail the same way. With the one-token fix the test passes. The same call without the leading separator returns sunflower (the audit's control run); it is not part of the repro.

Use the same test as the regression test.

cp $REPRO/TestWordBreakCombineLeadingSeparator.java $LUCENE/lucene/suggest/src/test/org/apache/lucene/search/spell/
cd $LUCENE && ./gradlew :lucene:suggest:test --tests TestWordBreakCombineLeadingSeparator -Ptests.seed=5EED1073 -Ptests.asserts=true
# also with -Ptests.asserts=false; nothing depends on the seed

TestWordBreakCombineLeadingSeparator > testLeadingSeparatorTerm FAILED
    java.lang.AssertionError: expected:<1> but was:<0>
        at junit@4.13.2/org.junit.Assert.fail(Assert.java:89)
        at junit@4.13.2/org.junit.Assert.failNotEquals(Assert.java:835)
        at junit@4.13.2/org.junit.Assert.assertEquals(Assert.java:647)
        at junit@4.13.2/org.junit.Assert.assertEquals(Assert.java:633)
        at org.apache.lucene.search.spell.TestWordBreakCombineLeadingSeparator.testLeadingSeparatorTerm(TestWordBreakCombineLeadingSeparator.java:47)
1 test completed, 1 failed

# the same with -Ptests.asserts=false, and on the 10.6.0 release jars with assertions on and off;
# with the fix: 1 test completed, 0 failed

• Contract. SEPARATOR_TERM: "Term that can be used to prohibit adjacent terms from being combined" (WordBreakSpellChecker.java:40). The method javadoc: "To prevent two adjacent terms from being combined (for instance, if one is mandatory and the other is prohibited), separate the two terms with SEPARATOR_TERM" (:136-137). It states the purpose, not a position rule, and says nothing about fields. Nothing in the javadoc, CombineSuggestion, the package docs or CHANGES (LUCENE-3523: "See Javadocs for usage") says all terms must share a field or that terms[0] chooses it.
• The code's own intent. The separator check at :170 runs for every i, 0 included, so a leading separator is skipped like any other; that shows the input is handled, not that it was designed for. Every other lookup uses the term's own field (:159, :278, :281, :312, :315).
• Tests. TestWordBreakSpellChecker never uses SEPARATOR_TERM; it passes unchanged with the fix.
• History. terms[0].field() is in the very first version of the class (642b882d5a5, LUCENE-3523, 2012) without a comment; the discussions on #4597 (9 comments) and SOLR-2993 never mention the lookup field or the separator's position.

The strongest "intended" case (the devil's-advocate verifier built it; the cross-model lenses rated it moderate): the class quietly assumes one field per call. suggestWordCombinations has no field parameter, CombineSuggestion carries no field, every upstream test passes one field, and Solr uses one field and emits the separator only between two tokens. Read that way, terms[0].field() is "the field of this call", and "separate the two terms" means a separator belongs between terms only, so a maintainer may reasonably call a leading separator unsupported.

Why it does not settle the claimed case: it holds for mixed-field input, which is why that case is not claimed. With a leading separator all real terms share one field; terms[0] is not a real term but the class's own constant, which has no field to share. The javadoc gives no position rule, the API accepts the input, and the loop's generic check skips a separator at index 0 like any other. The behaviour is undocumented and surprising: an element documented only to keep two neighbours apart silently turns off every combination when it stands in front.

One fix was built and validated in a scratch worktree of 645d4fe14a20; the other options are listed with their trade-offs. Workaround until a fix ships: drop a leading SEPARATOR_TERM before the call (or add separators only for i > 0, as Solr does), and add one to each of the returned originalTermIndexes() if they are mapped to the caller's own array, because they then count positions in the shortened array ([f:sun, f:flower] gives [0, 1]; the fixed class gives [1, 2] for [SEPARATOR_TERM, f:sun, f:flower]).

At WordBreakSpellChecker.java:203, use terms[i].field(). terms[i] is the left term of the run, which is never a separator (:170-172), so a leading separator no longer changes the field. For every one-field call without a leading separator the result is unchanged.

--- a/lucene/suggest/src/java/org/apache/lucene/search/spell/WordBreakSpellChecker.java
+++ b/lucene/suggest/src/java/org/apache/lucene/search/spell/WordBreakSpellChecker.java
@@ -200,7 +200,7 @@
           minFreq = Math.min(minFreq, origFreqs[j]);
         }
 
-        Term combinedTerm = new Term(terms[0].field(), combinedTermText);
+        Term combinedTerm = new Term(terms[i].field(), combinedTermText);
         int combinedTermFreq = ir.docFreq(combinedTerm);

Trade-offs

• It also changes mixed-field output: whenever a run starts after a terms[0] of another field, the run is now looked up in its left term's field. A run that crosses fields is still looked up in one field; the fix does not define what such a run should mean.
• Open PR #16847 edits the next line (:204, int to long), so one of the two needs a trivial rebase.

What the checks validated

• Gradle, JDK 25, scratch worktree of 645d4fe14a20: the test fails 1 of 1 on stock with assertions on and off; with the fix, the test plus TestWordBreakSpellChecker pass (2 suites, 6 tests) with assertions on and off; the whole :lucene:suggest:test passes with assertions on and off (32 suites, 237 tests, 7 skipped, 0 failures each); :lucene:suggest:check -x test passes (checkGoogleJavaFormat, ecjLint, forbiddenApis).
• The hunt: the repro and the independent reproducer's and the cross-model reviewers' own tests pass with the fix; in the bounded model the fix alone changes exactly the 416,768 calls whose result differs from the model's reference for this line, and no other.

In the inner loop, break when !terms[j].field().equals(terms[i].field()), as at a separator, so words of two fields are never joined.

Trade-offs

Changes mixed-field output, which has no documented expectation today: a separate design question, not part of the bug fix. Not validated.

Compute the field once before the loop from the first non-separator term.

Trade-offs

Same result as fix A for the claimed case and for one-field calls, and keeps "one field per call" explicit; slightly more code, and mixed-field calls still use one field. Not validated.

Say in the javadoc that the field of terms[0] is used for every combination.

Trade-offs

No behaviour change; the trap stays for callers that build separators in a loop.

Method: semi-formal (bounded translation validation of the real class against executable reference models, then executed JUnit tests).

• A hunt unit (spellchecker, dictionaries, word break), 2026-10-26: while building a bounded translation validation of suggestWordCombinations (400,000 queries, 0 mismatches), it excluded leading separators because the code looks combinations up in terms[0].field(), and witnessed that with a JUnit that failed on main. It recorded the lead without claiming it ("needs an unusual input").
• The adjudication (2026-11-02) re-ran the lead, reduced it to one public-API JUnit with one document, and compared the stock jar with four reference models on 2,283,840 calls (40 small random indexes, two fields, leading, inner and trailing separators, every SuggestMode, minSuggestionFrequency 0-3, maxChanges 1-2). The jar matches the code-semantics model on every call; against the model's reference for this line (each run looked up in its left term's field) 416,768 calls differ, and the fix alone changes exactly those; of the differing result entries, 29,716 are one-field combinations lost or mis-counted only because of a leading separator, and the rest are in mixed-field calls, which no contract covers. So it is a separate root cause from the minSuggestionFrequency lead at :208.
• Independent verifiers, cross-model reviews, an adjudication and an audit then re-ran it (see the checks below).