A TermInSetQuery with a term over 1000 bytes breaks the UnifiedHighlighter without WEIGHT_MATCHES

For a set of two or more distinct terms, TermInSetQuery#visit reports its terms through QueryVisitor#consumeTermsMatching with a supplier that builds Automata.makeBinaryStringUnion over all terms. That builder rejects any term longer than Automata.MAX_STRING_UNION_TERM_LENGTH (1000 UTF-8 bytes), a guard for its recursive construction. TermInSetQuery itself accepts such terms and searches with them, so the supplier throws IllegalArgumentException for the query's own term set. The UnifiedHighlighter calls the supplier when WEIGHT_MATCHES is off and MULTI_TERM_QUERY is on (the default), that is after withWeightMatches, withHighlightPhrasesStrictly or withPassageRelevancyOverSpeed is set to false, so highlight() throws instead of returning snippets; custom visitors that call the supplier fail too. The documented equivalent, a ConstantScoreQuery over a BooleanQuery of SHOULD TermQuerys, highlights the same input. Search, count and default-flag highlighting are unaffected. It came with LUCENE-9349 (8.6.0; in 8.6.0-8.11.4 the default highlighter was affected too), is on main and in 10.6.0, and is not fixed upstream.

// TermInSetQuery.java at 5adff78bb168 (excerpt, "// :NNN" notes added)
public void visit(QueryVisitor visitor) {                               // :178
  ...
  if (termData.size() == 1) visitor.consumeTerms(this, new Term(field, ...));      // :182-184
  if (termData.size() > 1)  visitor.consumeTermsMatching(this, field, this::asByteRunAutomaton); // :185-187
}
private ByteRunAutomaton asByteRunAutomaton() {                         // :192
  Automaton a = Automata.makeBinaryStringUnion(termData.iterator());   // :194, every term
  return new ByteRunAutomaton(a, true);
}

// StringsToAutomaton.java
if (current.length > Automata.MAX_STRING_UNION_TERM_LENGTH) {         // :267
  throw new IllegalArgumentException("This builder doesn't allow terms that are larger than " ...);
}
// Automata.java:47-51: MAX_STRING_UNION_TERM_LENGTH = 1000, "to ensure the stack doesn't overflow
// while building, since our algorithm currently relies on recursion"

1. UnifiedHighlighter.getHighlightComponents extracts automata when WEIGHT_MATCHES is not in the flags (UnifiedHighlighter.java:1170-1172); WEIGHT_MATCHES is dropped when withWeightMatches, withHighlightPhrasesStrictly or withPassageRelevancyOverSpeed is false (:518-527). withHandleMultiTermQuery(false) drops it too, but it also removes MULTI_TERM_QUERY, so no automata are extracted and the supplier is not called (by reading).
2. getAutomata (:1238-1251) visits the query only when MULTI_TERM_QUERY is in the flags (:1248-1250; on by default); for the two-term set, TermInSetQuery.visit calls consumeTermsMatching, and the highlighter's collector immediately calls automaton.get() (MultiTermHighlighting.java:83-85).
3. The supplier runs makeBinaryStringUnion over "short" and the 1001-byte term; StringsToAutomaton.add throws for the long term before adding it (StringsToAutomaton.java:266-273). No stack overflow, only the exception, which ends the whole highlight() call.
4. With default flags the supplier is never called: a TermInSetQuery counts as an unrecognized leaf under WEIGHT_MATCHES (UnifiedHighlighter.java:1185-1205), so Weight#matches does the work.

Within the limit, an exhaustive check over a small enumerated corpus (3,280,539 term sets, 9,712,633,037 membership checks) found no mismatch between the supplier's automaton and the term set, which supports that the defect is the length limit only.

• Who is hit. Code that highlights a TermInSetQuery with a UnifiedHighlighter whose flags lack WEIGHT_MATCHES but keep MULTI_TERM_QUERY (on by default), when the set has at least two distinct terms (duplicates are merged, TermInSetQuery.java:123-159) and one of them is over 1000 bytes. The query can come from new TermInSetQuery(...), KeywordField.newSetQuery or a BooleanQuery that contains one; the set-query field must pass the field matcher (the highlighted field, or another field under an accept-all matcher). Custom or downstream QueryVisitors that call the consumeTermsMatching supplier fail the same way (by reading; a plain visitor that calls supplier.get() is executed).
• Default vs opt-in. Since 9.0.0 (LUCENE-9431) the default highlighter is not affected; withWeightMatches(false), withHighlightPhrasesStrictly(false) and withPassageRelevancyOverSpeed(false) are documented, non-deprecated builder options. In 8.6.0-8.11.4 the default UnifiedHighlighter was affected.
• Scale. None: one document and one 1001-byte value. The value does not have to be in the index, so one stray long id or URL in a user-supplied list is enough. The limit counts UTF-8 bytes, not characters.
• Effect. The whole highlight() call throws, so no document gets snippets. No wrong result, hang, extra resource use or index change; search, count, explain and Weight#matches of the same query work.
• How likely. Rare: no report of this path was found in the more than six years since 8.6.0.

Public API only (TermInSetQuery, UnifiedHighlighter.builder); the package org.apache.lucene.search.uhighlight is only for locality, and the file belongs in lucene/highlighter/src/test/org/apache/lucene/search/uhighlight/. The test framework picks the directory and codec from the seed (gradle reported codec=Asserting(Lucene104)), which does not matter here: the exception comes from building the visitor's automaton. It passes :lucene:highlighter:checkGoogleJavaFormat, :lucene:highlighter:ecjLintTest and :lucene:highlighter:forbiddenApisTest. It does not compile before 9.1.0 (the builder API).

Regression test: this same test. It passes with fix A, with assertions on and off.

# gradle, main 5adff78bb168, seed DEADBEEF, assertions on; -Ptests.asserts=false gives the same failure
TestTermInSetQueryLongTermHighlight > testHighlightWithoutWeightMatches FAILED
    java.lang.IllegalArgumentException: This builder doesn't allow terms that are larger than 1000 UTF-8 bytes, got [78 78 78 ... 78]
        at org.apache.lucene.util.automaton.StringsToAutomaton.add(StringsToAutomaton.java:268)
        at org.apache.lucene.util.automaton.StringsToAutomaton.build(StringsToAutomaton.java:260)
        at org.apache.lucene.util.automaton.Automata.makeBinaryStringUnion(Automata.java:688)
        at org.apache.lucene.search.TermInSetQuery.asByteRunAutomaton(TermInSetQuery.java:194)
        at org.apache.lucene.search.uhighlight.MultiTermHighlighting$AutomataCollector.consumeTermsMatching(MultiTermHighlighting.java:84)
        at org.apache.lucene.search.TermInSetQuery.visit(TermInSetQuery.java:186)
        at org.apache.lucene.search.uhighlight.MultiTermHighlighting.extractAutomata(MultiTermHighlighting.java:46)
        at org.apache.lucene.search.uhighlight.UnifiedHighlighter.getAutomata(UnifiedHighlighter.java:1249)
        at org.apache.lucene.search.uhighlight.UnifiedHighlighter.getHighlightComponents(UnifiedHighlighter.java:1172)
        ...
        at org.apache.lucene.search.uhighlight.UnifiedHighlighter.highlight(UnifiedHighlighter.java:748)
        at org.apache.lucene.search.uhighlight.TestTermInSetQueryLongTermHighlight.testHighlightWithoutWeightMatches(TestTermInSetQueryLongTermHighlight.java:66)

# [78 78 ... 78] is the 1001-byte term ("x" is 0x78), shortened here
# release jars, plain JUnit runner, seed DEADBEEF: 10.6.0 (assertions on and off) and 9.12.3: the same
# exception; 9.6.0: IllegalArgumentException: input automaton is too large: 1001 (Operations.isFinite)
# with fix A: BUILD SUCCESSFUL, assertions on and off

• The contract. QueryVisitor#consumeTermsMatching: "a supplier for a runnable automaton defining which terms match" (QueryVisitor.java:46). TermInSetQuery's class javadoc: it "behaves like a ConstantScoreQuery over a BooleanQuery containing only SHOULD clauses" (TermInSetQuery.java:42-45), and that form highlights the same input (the javadoc is about matching and scoring, so this is an observed control, not a documented highlighting promise). TermInSetQuery, KeywordField.newSetQuery, QueryVisitor and UnifiedHighlighter#highlight document no term-length limit; highlight() documents only IOException and an IllegalArgumentException for a field without offsets (UnifiedHighlighter.java:743-746). The limit is documented only on the builder constant, whose javadoc names makeStringUnion(Iterable), not this query (Automata.java:47-50).
• The tests. TestTermInSetQuery.testVisitor (TestTermInSetQuery.java:508-550) asserts that the supplier's automaton accepts every term of the query, with short terms only. TestUnifiedHighlighter.testQueryWithLongTerm (TestUnifiedHighlighter.java:1766-1791) asserts that a BooleanQuery with a 1001-byte TermQuery highlights. No highlighter test uses a TermInSetQuery.
• The history. #11808 (188a78d7697, 2022) filters long terms on the MemoryIndexOffsetStrategy path (MemoryIndexOffsetStrategy.java:64-69); its commit message says "Rather than throwing a confusing exception, we can instead filter out the long terms". 11436a848cb (2023) made a test helper fall back to Operations.union for a too-long term. asByteRunAutomaton has no such guard. LUCENE-9349, which introduced the supplier, and #12320, which moved it to the string-union builder, do not discuss term length.

The strongest case for "intended": consumeTermsMatching and Automata are @lucene.experimental; the builder limit is deliberate and tested (TestStringsToAutomaton.testLargeTerms); turning WEIGHT_MATCHES off selects the older automaton path; and #11808's commit message says "Very long terms are likely to be junk input in any case".

Why it fails: none of this is stated at TermInSetQuery, QueryVisitor or the highlighter; the query accepts and matches the term; the documented equivalent highlights; and #11808 answered the same exception on the TermQuery path by filtering, not by calling it intended. The cross-model reviewers rated the intended case moderate, the devil's-advocate check and the adjudication weak; every reviewer agreed on the root cause. A maintainer may still reasonably read it as a gap that #11808 left open (same highlighter, same exception, another path and fix site).

Option A is validated; B and C are alternatives a maintainer may prefer.

The diff is +24/-2 in TermInSetQuery.java only; makeBinaryStringUnion and StringsToAutomaton are unchanged. The terms still stream into makeBinaryStringUnion, through a wrapping iterator in asByteRunAutomaton that skips terms over the limit; those are built with Automata.makeBinary(term), and only when there is at least one, the string union and the long terms are combined with Operations.union and Operations.determinize(..., Integer.MAX_VALUE).

// in asByteRunAutomaton(): terms over the limit skip the string union
if (term.length <= Automata.MAX_STRING_UNION_TERM_LENGTH) {
  return term;                                   // to makeBinaryStringUnion
}
longTerms.add(Automata.makeBinary(term));
...
if (longTerms.isEmpty() == false) {
  longTerms.add(a);
  a = Operations.determinize(Operations.union(longTerms), Integer.MAX_VALUE);
}
return new ByteRunAutomaton(a, true);

Trade-offs

Sets without a long term take the old path unchanged. Sets with one get a non-minimal automaton with roughly one extra state per byte of each long term; the unbounded determinize limit is safe because the result has at most the union's states plus those (reasoned from the construction, not benchmarked). It does not remove the builder limit for other callers.

What the checks validated

Gradle in a scratch worktree of 5adff78bb168, seed DEADBEEF: the repro fails stock and passes with the fix, assertions on and off; the fix passes :lucene:core:checkGoogleJavaFormat, ecjLintMain and forbiddenApisMain; with the fix TestTermInSetQuery, TestQueryVisitor, TestKeywordField and a 3-test probe (a 32766-byte term with a short one, a set of only long terms, a 1000-byte term on the unchanged path; each checks accepted and rejected terms) pass (4 classes, 27 tests, 1 skipped, 0 failures), and the whole org.apache.lucene.search.uhighlight package passes (14 classes, 471 tests, 0 failures). Targeted suites, not a full build.

The highlighter cannot see inside the supplier, so it would have to special-case TermInSetQuery (for example via getBytesRefIterator()) or catch the exception.

Trade-offs

Fixes only the highlighter (other visitors still fail) and drops the long term from highlighting, as #11808 does.

Remove the recursion in the string-union builder so it accepts terms up to the index limit. Similar recursion limits on long legal input were lifted in #11813 (the recursive Operations.isFinite was taken out of production code on main and kept as a test utility) and #12286 (topoSort made iterative).

Trade-offs

Removes the cause for every caller (the #11808 filter and the test-helper workaround become unnecessary) but changes a core automaton builder. Note: the TODO above asByteRunAutomaton points at GH#12176; draft #12312 would build the same union in the constructor, where this limit would reject the query itself unless A or C lands first.

Method: semi-formal (bounded boundary-value translation validation on the stock main jar).

• The matrix. Term byte length 999 / 1000 / 1001 / 4096 / 32766, in ASCII, 3-byte and 4-byte UTF-8; a TermInSetQuery with 1, 2 or 50 terms, KeywordField.newSetQuery and a BooleanQuery containing the set; each through search, count, explain, Weight#matches, a plain QueryVisitor that calls the supplier, and the UnifiedHighlighter with default flags and with each of the three flags that drop WEIGHT_MATCHES; offsets from postings, term vectors and re-analysis.
• Result. Every cell with a term over 1000 bytes and at least two terms threw IllegalArgumentException from the supplier and from the three non-WEIGHT_MATCHES highlighters; 999 and 1000 bytes never threw; search, count, explain and matches never threw.
• The language within the limit. An exhaustive check of the supplier's automaton against the term set (3,280,539 sets, 9,712,633,037 membership checks, 0 mismatches; a deliberately broken reference was detected, so the check is not vacuous) showed that the defect is the limit only.
• The cause was then read from TermInSetQuery and StringsToAutomaton, and a public-API JUnit reproduced it on main (assertions on and off) and on releases back to 9.1.0 (assertions on).