Monitor's RegexpQueryHandler silently skips regexp queries on short words that contain an emoji or other supplementary character

RegexpQueryHandler is the optional handler that lets a Monitor's TermFilteredPresearcher pre-select RegexpQuerys: it indexes each regexp under its longest literal plus XX and runs each document field that is not in its excludedFields through SuffixingNGramTokenFilter, which emits each short token and its substrings plus XX (skipping substrings already emitted in the same field). The filter measures substrings in code points but starts and resets their size from the token's UTF-16 length, so for any token that contains a supplementary character (an emoji, a CJK Extension B ideograph, a letter of Adlam, Osage or Gothic) the first bound check fails and the token gets no substrings at all. A regexp query that only one of them could select is not selected: /hel.*/ is not reported for hello😀, with 0 queries run and no error, while a Monitor without filtering reports it. It affects tokens of at most 30 UTF-16 chars (the default maxTokenSize) unless another token of the field supplies the substring, a token of the field is longer than 30 chars (it emits the wildcard token), or another document of the same match batch does either; also under StandardAnalyzer for words in supplementary-plane scripts; it never causes false matches and leaves term queries and the default presearcher alone. It is in every lucene-monitor release, 8.2.0 to 10.6.0 (10.6.0 by reading), and on main; the fix is three lines.

• Monitor (lucene/monitor): stores queries and reports which of them a document matches. Its presearcher turns the document's terms into a query over the stored queries, so that only queries that can match are run (package-info.java:55-63).
• RegexpQueryHandler: an optional CustomQueryHandler for TermFilteredPresearcher. It indexes /hel.*/ under its longest literal plus the suffix, helXX, and under the wildcard token __WILDCARD__ (RegexpQueryHandler.java:96-114), and wraps each document field not in its excludedFields in SuffixingNGramTokenFilter (:90-93; defaults XX, 30, __WILDCARD__ at :41-47). The query is selected if the document yields helXX or __WILDCARD__.
• Supplementary character: a code point above U+FFFF (an emoji, a CJK Extension B ideograph, Adlam, Osage, Gothic letters). Java stores it as two chars, a surrogate pair, so such a word has more UTF-16 chars than code points.

1. TermFilteredPresearcher.buildQuery (TermFilteredPresearcher.java:100-113, final) reads each field's distinct terms as a TermsEnumTokenStream (:110) and wraps it with every handler (:112). That stream carries only a CharTermAttribute (TermsEnumTokenStream.java:27-30), so the filter's keyword pass-through (:84) never fires in the Monitor.
2. SuffixingNGramTokenFilter.incrementToken (SuffixingNGramTokenFilter.java:76-136) first returns the token itself (:96), after setting curTermLength = UTF-16 length (:87), curCodePointCount (:88) and curGramSize = curTermLength (:89).
3. Each substring must pass (curPos + curGramSize) <= curCodePointCount (:110) and is sliced with Character.offsetByCodePoints (:112-114), both in code points; the size is reset to curTermLength - curPos (:108) and the suffix check uses curTermLength - curPos (:117), in UTF-16 chars.
4. For hello😀: 7 chars, 6 code points, so the first check is 0 + 7 <= 6, false. The filter drops to :134 and takes the next token, having emitted only hello😀. One surrogate pair anywhere in a token is enough; not even XX is emitted.
5. The document yields neither helXX nor __WILDCARD__, so /hel.*/ is not selected and Monitor.match never runs it (Monitor.java:224, :294-299).
6. Tokens longer than 30 chars take the wildcard path (:99-104), which every regexp query collects (RegexpQueryHandler.java:113), so they are safe (executed: 30 chars missed, 31 reported), and any such token in the same field hides the miss. The presearcher query is built once per match(Document[]) batch (Monitor.java:218-225), so a token of another document in the batch can also select the query (executed).

Executed on main (-ea and -da), one token per document: hello gives 16 terms including helXX and the query runs; hello😀 gives 1 term and 0 queries run; with the fix hello😀 gives 22 terms and the query matches; a😀 gives [a😀] instead of {a😀, a😀XX, aXX, 😀XX, XX}.

• Who is exposed. Monitors built with TermFilteredPresearcher or MultipassTermFilteredPresearcher (which inherits the final buildQuery; executed with the same misses) plus RegexpQueryHandler, that register RegexpQuerys on fields not in its excludedFields. Code search found no public user outside copies of Lucene's own source; Solr's open Monitor integration PR passes no custom handlers.
• Default vs opt-in. The handler is opt-in; the default presearcher does not use this filter and reported every probe document. Inside the opt-in configuration nothing special is needed: new RegexpQueryHandler(), stock analyzers, assertions on or off.
• Which text. Executed: WhitespaceAnalyzer with hello😀, say hello𠀀 now (U+20000), x😀hello (/.*hello/) or the Gothic word 𐌰𐌱; KeywordAnalyzer with great product 😀; StandardAnalyzer with short Adlam or Osage words, hello𝐀 (U+1D400) or hello plus a variation selector (U+E0100) or tag character (U+E0041), default-ignorable code points that are normally not rendered. StandardAnalyzer splits an emoji or a CJK ideograph off a Latin word, so hello😀 is found there.
• Bounds. Only tokens of at most maxTokenSize UTF-16 chars (default 30); another token of the same field that yields the needed substring hides the miss, and so does any token of the field longer than 30 chars (it emits __WILDCARD__) or such a token in another document of the same match(Document[]) batch (executed; a token in another field does not help); term queries are unaffected because the token itself is always emitted.
• Scale. None: one token of two code points is enough. Exhaustive check of the stock jar over all 1,560 inputs of 1-2 tokens of 1-3 code points over {a, b, U+1F600}: 1,350 miss required substrings (each input on its own), exactly those with U+1F600; 0 of the 210 BMP-only inputs; 0 with the fix.
• Damage. Silent false negatives for regexp queries only: no error, 0 queries run. Never false matches (0 in an exhaustive probe of 1,764 documents x 10 queries, where 2,478 of 4,520 true pairs were missed), no hang, less work rather than more.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: queries run: 0 expected:<1> but was:<0>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:647)
       org.apache.lucene.monitor.TestRegexpQueryHandlerSupplementaryChar.testRegexpMatchesTokenWithEmoji(TestRegexpQueryHandlerSupplementaryChar.java:39)

• Contracts. RegexpQueryHandler class javadoc: "A query handler implementation that matches Regexp queries by indexing regex terms by their longest static substring, and generates ngrams from Document tokens to match them" (RegexpQueryHandler.java:29-31). SuffixingNGramTokenFilter constructor: "@param maxTokenLength tokens longer than this will not be ngrammed" (SuffixingNGramTokenFilter.java:60), the filter's only documented exclusion; the handler also documents excludedFields, "a Set of fields to ignore when generating ngrams" (RegexpQueryHandler.java:64, applied at :91), and the filter passes keyword-marked tokens through (:84, undocumented; never the case in the Monitor path). None covers short tokens with supplementary characters. Package doc: queries that match the document disjunction "are then run against the document. In this way, the Monitor can avoid running queries that have no chance of matching" (package-info.java:60-61).
• The code's own unit. :88 computes curCodePointCount, :110 bounds substrings by it and :112-114 slice with Character.offsetByCodePoints (whose curTermLength argument is the char-array bound, correctly in chars); of the gram arithmetic, only :89, :108 and :117 use the UTF-16 length. :99 also compares the UTF-16 length with maxTokenLength, which only over-selects and is a separate question (option "maxlength-codepoints"). The mix is not an alternative design: it makes the first check fail for every token with a surrogate pair.
• Tests. TestSuffixingNGramTokenizer and TestRegexpQueryHandler use ASCII only; no upstream test covers non-BMP text. TestWildcardTermPresearcher runs RegexpQueryHandler through the shared PresearcherTestBase suite (TestWildcardTermPresearcher.java:25, :93), so it is a supported configuration.
• History. The lines came with the module (LUCENE-8766, 251dbe7cea8, 2019), from the donated Luwak code, whose master still has them; neither LUCENE-8766's 28 comments nor its import PR discusses Unicode.
• Precedent. LUCENE-5042: "The n-gram and edge n-gram tokenizers and filters now correctly handle supplementary characters" (lucene/CHANGES.txt:13522-13524); LUCENE-5269: "Fix bug in NGramTokenFilter where it would sometimes count unicode characters incorrectly" (:13215-13216); #15899/#15900, TruncateTokenFilter (10.5.0, :1182-1184), where a maintainer wrote: "We modified all other similar filters in the past (like ngrams), they all work correctly." Missed Monitor matches fixed as bugs: LUCENE-9791 (:5370-5372), GITHUB#14886 (:1743-1746). The UTF-16-versus-code-point pattern itself is not new.

The strongest case for "intended or best effort" (built by the devil's-advocate verifier; rated weak by it, the independent reproducer, the GPT-5.6 "works as designed" reviewer and the adjudication): the package doc says RegexpQueryHandler "gives an example of a different method of indexing automaton-based queries" (package-info.java:93-95); it is opt-in; the filter is package-private; the tests are ASCII-only and nothing promises Unicode support; and RegexpQuery itself is tagged @lucene.experimental (RegexpQuery.java:40).

Why it fails: these limit reach, not intent. The handler is public API with public defaults and constructors, has no @lucene.experimental tag and runs in the upstream presearcher test suite; the experimental tag on RegexpQuery marks an API that may change, not licence for the presearcher to drop a query the document matches (under Presearcher.NO_FILTERING the same query matches); the regexps in the repro (a literal prefix plus .*) are the shape the handler's own tests cover; every short token with a surrogate pair gets no substrings at all, which is not an edge of regexp syntax; and the filter's own code counts code points, as the project expects of n-gram filters (#15899).

One fix was built and validated in a scratch worktree at ea57360912ed (gradle, JDK 25; removed afterwards); an alternative was run against this doc's test. Workaround in every affected release: do not use RegexpQueryHandler (the default presearcher selects every regexp query), or use a copy of the handler with the fix.

Replace curTermLength by curCodePointCount at :89, :108 and :117, so the size uses the unit the bound check (:110) and the slicing (:112-114) already use. BMP tokens are unchanged, since both units agree. Diffs: codepoint-fix.diff (code only) and fix-with-test.diff (with both tests; applies cleanly at ea57360912ed).

@@ -86,7 +86,7 @@ final class SuffixingNGramTokenFilter extends TokenFilter {
         curCodePointCount = Character.codePointCount(termAtt, 0, termAtt.length());
-        curGramSize = curTermLength;
+        curGramSize = curCodePointCount;
@@ -105,7 +105,7 @@ final class SuffixingNGramTokenFilter extends TokenFilter {
         ++curPos;
-        curGramSize = curTermLength - curPos;
+        curGramSize = curCodePointCount - curPos;
@@ -114,7 +114,7 @@ final class SuffixingNGramTokenFilter extends TokenFilter {
-        if ((curGramSize == curTermLength - curPos)
+        if ((curGramSize == curCodePointCount - curPos)
             && !seenSuffixes.add(termAtt.subSequence(0, termAtt.length()))) {

Trade-offs

• Tokens with supplementary characters now get their substrings, so the presearcher query grows for them; the per-token count stays within the bound BMP tokens already reach (1 + c(c+1)/2 + 1 for c code points): a 30-char token of 15 different emoji gives 122 terms, a 30-char BMP token already 457.
• No performance benchmark was run (only tokens with surrogate pairs change); not reviewed by a committer.

What the checks validated

• Stock: the doc's 3 tests and the new tokenizer case fail (4 of the 8 tests in the two classes), assertions on and off; with the fix all 8 pass, assertions on and off (gradle).
• The whole monitor module with the fix: 33 classes, 185 tests (181 upstream + 4 new), 0 failures, 1 skipped, assertions on and off.
• :lucene:monitor:check -x test passes (google-java-format, ecjLint, forbiddenApis).
• The harness with the fixed class shadowed: the doc's test 3 run, 0 failed, -ea and -da. The class is byte-identical to the hunt's, with which the finder's, the independent reproducer's, the GPT-5.6 reviewer's and the devil's advocate's tests pass, the exhaustive check finds 0 missing substrings, and 8.2.0 passes.

:99 compares the UTF-16 length with maxTokenLength, so a token of 16 emoji (32 chars) takes the wildcard path. That only over-selects (the query runs and is checked), so it is safe; the javadoc ("the maximum length of an input token") names no unit.

Trade-offs

Not needed for correctness; the tokens of 31-60 UTF-16 chars that have at most 30 code points would be ngrammed instead of wildcarded (more terms, fewer queries run); a 32-char token with one surrogate pair has 31 code points and would still be wildcarded. Not validated.

if (curTermLength > maxTokenLength || curTermLength != curCodePointCount) at :99 (option-wildcard.diff).

Trade-offs

Every regexp query on that field then runs for every document batch with such a token, which defeats the handler's filtering for emoji-rich text; the recommended fix is as small and keeps the filtering.

What the checks validated

Both Monitor tests of this doc pass; testShortTokenIsNgrammed fails by design (it asserts the substrings; got [a😀, __WILDCARD__]).

Not recommended.

Trade-offs

Leaves silent misses for ordinary text in the opt-in configuration.

Method: semi-formal (a code-reading hypothesis witnessed by bounded-exhaustive translation validation), a hunt unit (monitor and presearcher, 2026-10-26).

• While modelling the document side of the presearcher, the finder (an AI tool) read SuffixingNGramTokenFilter and saw the UTF-16 length and the code-point count mixed in one loop. No formal tool produced the first signal.
• It then ran every input of 1-2 tokens of 1-3 code points over {a, b, U+1F600} (1,560 inputs) through RegexpQueryHandler.wrapTermStream on the stock jar against an exact reference (every code-point substring plus XX): 1,350 inputs miss required substrings, exactly those that contain U+1F600, and 0 of the 210 BMP-only inputs; with the fix, 0.
• It minimised this into a public-API JUnit on the stock jars. Then: a refute review by an independent reviewer from another model family (OpenAI GPT-5.6, reading only), an independent reproducer, a GPT-5.6 "works as designed" review whose test design was executed, a devil's advocate with stock analyzers, an adjudication, a release sweep, a prior-report search, a security screen, a credibility audit, and this doc's own test and fix validation.