HTMLStripCharFilter drops the text after a tag whose name only begins with script or style (since 10.5.0)

HTMLStripCharFilter has dedicated start-tag rules for br, script and style (HTMLStripCharFilter.jflex:658, :673, :682). After a script or style start tag the scanner drops every character up to the next </script> or </style> end tag, or to the end of the input. Until 10.4.0 these rules required whitespace (or = for br) after the name. #15756, the fix for #15754 (attribute values ending in =), moved that whitespace into the attribute macros and dropped it from the three rules, so the text after the name may now start with any character except < and > (or with a server-side include). <scripts> then matches the script rule with one character more than the generic element rule, and JFlex's longest-match rule picks it: <scripts>hello</scripts> world strips to an empty string instead of \nhello\n world, a tokenizer after the filter emits no token, and no exception is raised. Custom elements such as <style-guide> or <script-editor> and XML-like names such as <stylesheet> or <styleUrl> lose the text the same way; with escapedTags, names that begin with br (<brand>, <bridge>) are escaped or stripped against the configured set. Released in 10.5.0 to 10.6.0 (10.6.0 by reading) and on main. The fix requires a delimiter after the name again (whitespace, / or an attribute); with it, all of :lucene:analysis:common:test passes (1,914 tests, 18 skipped, 0 failures).

• HTMLStripCharFilter: a CharFilter in lucene/analysis/common that removes HTML markup and decodes character references before tokenization. Its scanner is generated by JFlex from HTMLStripCharFilter.jflex. An unknown element's start and end tags are each replaced by a newline and its text is kept; the content of <script> and <style> elements is removed.
• Longest match: JFlex picks the rule that consumes the most input. When two rules can match a start tag, the longer match wins.
• escapedTags: an optional constructor argument; "Tags in this set (both start and end tags) will not be filtered out."

1. The three rules (HTMLStripCharFilter.jflex:658, :673, :682, right after <):

   \s* [bB][rR] ( "="\s* {OpenTagContent} | {OpenTagContent} )? \s* "/"? ">"
   \s* [sS][cC][rR][iI][pP][tT] ( {OpenTagContent} )? \s*  ">"      -> yybegin(SCRIPT)
   \s* [sS][tT][yY][lL][eE] ( {OpenTagContent} )? \s* ">"           -> yybegin(STYLE)

1. OpenTagContent (:115) is ( {EventAttribute} | [^<>] | {ServerSideInclude} | {OpenTagAttribute} )*. Its [^<>] alternative matches any character except < and >, letters included, so nothing has to separate script from what follows.
2. For <scripts> two rules compete: the generic {Name} rule (:703-711) matches scripts (7 characters), the SCRIPT rule matches script + s + > (8). The longer match wins and the scanner enters the SCRIPT state.
3. In SCRIPT (:860-885) only <!-- and the end tag </script> (:862: the exact name, whitespace allowed) are special; every other character is skipped (:884), </scripts> included. At the end of the input the rest is excluded (:275-296). STYLE (:887-912) is the same with </style>.
4. The br variant: <brand> matches the BR rule (6 characters) instead of {Name} (5). escapedTags is checked by exact name: BR, SCRIPT and STYLE become flags (:181-197), other names are looked up only by the generic {InlineElment} and {Name} rules (:591-608, :694-711); the BR, SCRIPT and STYLE rules check only their flags. So with {brand} the start tag is replaced by a newline while </brand> is kept; with {br}, <bridge> is kept; with {script}, <scripted> is echoed and the rest dropped. Without escapedTags the BR variant is invisible: both rules emit a newline.
5. The end-tag rules match only the exact names: </br> (:576), </script> (:862), </style> (:889).
6. Until 10.4.0 the three rules were ( \s+ {OpenTagContent} )? (script, style) and ( ( "="\s* | \s+ ) {OpenTagContent} )? (br), unchanged since the 2012 JFlex rewrite. #15756 added a leading \s+ to EventAttribute and OpenTagAttribute (:113-114) and removed it from these rules, which also removed the only thing that ended the tag name.

Not affected: real <script>, <style> and <br> tags with or without attributes, and the controls with other names (<scrapes>, <stale-guide>, <bland>, <pizza hut>), executed; from the grammar, any name that does not begin with these words goes through the generic rules as before. A KML document loses everything after <styleUrl> in the same way. Output strings: the PrefixProbe on main and 10.4.0 and the fact check's own probe on main, 10.5.2 and 10.4.0 (10.5.0 and 10.5.1 have the same grammar as 10.5.2; there only the standalone JUnit was run). The token result: the standalone JUnit with a StandardTokenizer and the fact-check probe.

• Who is exposed. Users of HTMLStripCharFilter directly, through the htmlStrip SPI factory (HTMLStripCharFilterFactory.java:42, e.g. in a CustomAnalyzer), or through a downstream html_strip char filter built on it. No Lucene analyzer adds it by default: opt-in, but a stock component; the script/style variant needs no setting.
• What it takes. A start tag whose name begins with script or style and goes on with a name character: hyphenated custom elements (<style-guide>, <script-editor>, <style-switcher>), XML-like names (<scripts>, <styles>, <stylesheet>, KML's <styleUrl>), or angle-bracket placeholders in plain text (<scripted hut>). No standard HTML element has such a name, so pages that use only standard elements are not hit, but valid custom-element markup is; how often these shapes occur in real content was not measured. The br variant needs escapedTags.
• What the damage looks like. Text after the tag is missing from the char-filter output and so from the index, up to the next real </script> or </style> end tag, or to the end of the field value if none follows. No exception, with assertions on or off; whatever came before the tag is indexed normally, so the document looks fine.
• Scale. Pure scanner logic, deterministic and independent of index size.
• Versions. 10.5.0, 10.5.1, 10.5.2, 10.6.0 (by reading) and main; 10.4.0 and older are not affected.
• How likely. Rare: the trigger is an unusual element name, and no report was found since 10.5.0 shipped (tag commit dated 2026-06-19).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: expected:<[
hello
 world]> but was:<[]>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.analysis.charfilter.TestHTMLStripScriptPrefixName.testElementNameStartingWithScript(TestHTMLStripScriptPrefixName.java:32)

• The design required a delimiter. From the 2012 JFlex rewrite (LUCENE-3690) through 10.4.0 the rules were ( \s+ {OpenTagContent} )? (script, style) and ( ( "="\s* | \s+ ) {OpenTagContent} )? (br); 9.12.3, 10.0.0 and 10.4.0 pass the test.
• #15756 was about something else. Its title, body, commit message, 4 comments, 2 reviews, CHANGES entry ("GITHUB#15754: Fix HTMLStripCharFilter to prevent tags from incorrectly consuming subsequent HTML when an attribute value ends with an equals sign.", CHANGES.txt:1160-1161, 10.5.0 Bug Fixes) and its four tests are all about attribute values ending in =. Dropping \s+ from the rules was the mechanical counterpart of moving it into the attribute macros.
• The grammar's own handling of every other name. An unknown element goes through {Name}: its tags become newlines and its text is kept. The end-tag rules for these three names match only the exact names (HTMLStripCharFilter.jflex:576, :862, :889).
• escapedTags. "Tags in this set (both start and end tags) will not be filtered out." (HTMLStripCharFilter.jflex:176-177).
• Unknown tags keep their text. In LUCENE-5943 (apache/lucene#7005, closed "Not A Problem" in 2014) the reporter's I love <pizza hut> so much came out as the tokens i, love, so, much: only the tag was removed. The filter's original author (sarowe) replied that angle brackets mark tags, which the filter strips, and confirmed with a test that I love <pizza hut> strips to I love \n. I love <pizza hut> so much still strips to I love \n so much on main, but I love <scripted hut> so much now loses so much.
• HTML. In the HTML tokenizer a tag name ends only at whitespace, / or >, so <scripts> is a scripts element, not a script element. The oracle here is the grammar's own {Name} handling and 10.4.0, not WHATWG equivalence.
• No test pins the prefix behaviour. No upstream test has a start tag named script, style or br followed by a name character; the existing ones (<script attr= bare>, <style type=...>, <BR class='whatever'>) all have whitespace after the name.

The strongest case for "intended": (a) the class only "attempts to strip out HTML constructs" (HTMLStripCharFilter.jflex:31), so it is best effort; (b) #15756 was reviewed and merged, so the new rules could be deliberate; (c) such element names are not standard HTML; (d) dropping too much is safer than letting script text through.

Why it fails: (a) best effort does not explain a behaviour change on markup the grammar handled the same way for 14 years; (b) nothing in #15756 discusses tag names; (c) the grammar accepts any XML-style name through {Name}, hyphenated custom elements are valid HTML element names, and LUCENE-5943 shows the intended handling of unknown tags; (d) the script and style states exist to remove script and style content, and a real <script> still matches as before; what is lost here is ordinary text, often the rest of the field. The escapedTags promise is broken outright. Every verifier rated the intended-behaviour case weak.

Two grammar options were applied in a scratch worktree of a9779ef3c193, each with the scanner regenerated by ./gradlew :lucene:analysis:common:regenerateHTMLStripCharFilter (JFlex 1.9.1; it also updates the checksum file), and run against TestHTMLStripCharFilter (with the five new tests) and TestHTMLStripCharFilterFactory, assertions on and off.

Add a macro AfterTagName = ( \s | "/" | {EventAttribute} | {OpenTagAttribute} ) {OpenTagContent} and use it instead of the bare {OpenTagContent} in the BR, SCRIPT and STYLE start-tag rules (HTMLStripCharFilter.jflex:658, :673, :682); keep the BR rule's "="\s* {OpenTagContent} alternative. Then regenerate HTMLStripCharFilter.java. Longer names then fall through to the generic {Name} rule like every other element.

+// Content after the BR, SCRIPT or STYLE tag name: it must start at a tag-name delimiter
+// (whitespace or '/', or an attribute, which starts with whitespace), so that a longer name
+// such as "scripts" or "brand" is not taken for one of these tags.
+AfterTagName = ( \s | "/" | {EventAttribute} | {OpenTagAttribute} ) {OpenTagContent}
...
-  \s* [bB][rR] ( "="\s* {OpenTagContent} | {OpenTagContent} )? \s* "/"? ">" {
+  \s* [bB][rR] ( "="\s* {OpenTagContent} | {AfterTagName} )? \s* "/"? ">" {
-  \s* [sS][cC][rR][iI][pP][tT] ( {OpenTagContent} )? \s*  ">" {
+  \s* [sS][cC][rR][iI][pP][tT] ( {AfterTagName} )? \s*  ">" {
-  \s* [sS][tT][yY][lL][eE] ( {OpenTagContent} )? \s* ">" {
+  \s* [sS][tT][yY][lL][eE] ( {AfterTagName} )? \s* ">" {

Trade-offs

• Keeps #15756's other change on purpose: <script/> starts a script element (as in HTML, where / does not close a script element; 10.4.0 kept the text after it). So it is "restore the delimiter after the name, also accepting /", not a revert.
• Real tags with attributes, <br/> and a > inside a quoted attribute value behave as on main (probe); the BR rule's = alternative is unchanged.
• The hunt's bounded-exhaustive harness with this fix: 0 mismatches in 655,889,580 runs (every sequence of up to 4 of 102 items: 64 base, 14 attribute and 24 prefix-name items; not independently re-run). Over all 551,880 strings of up to 4 of the harness's 27 symbols it changes main's output on 200 inputs (escapedTags off) and 298 (on), every one of them with <script, <style or <br directly followed by a non-delimiter (re-run by the fact check). Bounded evidence, not a proof.

What the checks validated

• The five new tests pass; TestHTMLStripCharFilter (41 tests) and TestHTMLStripCharFilterFactory (7) pass with assertions on and off: 48 tests, 0 failures (stock: 4 failures).
• All of :lucene:analysis:common:test with assertions on: 339 classes, 1,914 tests, 18 skipped, 0 failures, 0 errors.
• :lucene:analysis:common:check -x test passes (generated-source checksum checks, checkGoogleJavaFormat, ecjLint, forbiddenApis).
• The scanner regenerated by the build is byte-identical to the finder's (428,370 bytes; stock 428,378).
• A public-API probe with this scanner gives 10.4.0's output for every prefix input and main's output for every real-tag input (<script/>, <br title="a>b">, an escaped script tag with > in an attribute).

( \s+ {OpenTagContent} )? for script and style, ( ( "="\s* | \s+ ) {OpenTagContent} )? for br: exactly the reverse of #15756's three rule edits.

Trade-offs

Smallest diff, but not 10.4.0's behaviour: #15756 also moved the leading \s+ into the attribute macros, so the rule's own \s+ eats the space an attribute needs, and a > inside a quoted value ends the tag. It also brings back 10.4.0's handling of <script/>. Not recommended.

What the checks validated

• The four prefix tests pass, and so do the 36 existing TestHTMLStripCharFilter tests and the 7 factory tests; the guard testScriptStyleAndBrTagsStillMatch fails with assertions on and off (48 tests, 1 failed): a<script/>f();</script>b gives a\nf();\nb.
• The probe shows a regression the existing tests miss: one<br title="a>b">two gives one\nb">two (main and 10.4.0: one\ntwo), and with escapedTags={script} the start tag <script title="a>b"> is echoed as <script title="a>.

Route these names through the generic {Name} rule and switch state in its action by exact name, as escapedTags already does.

Trade-offs

A larger restructuring of the grammar; not prototyped.

Method: bounded-exhaustive translation validation of the scanner (registry tier semi-formal: a hypothesis formed while modelling, then witnessed by the tool).

• The harness runs every sequence of up to 4 of 64 well-formed constructs, with 3 escapedTags configurations and whole and 1-character Readers, on the stock jar, and compares the exact output and the per-character source offsets with values written from the grammar's documented replacement policy. On those items: 0 mismatches.
• While writing the item table, the finder read the BR, SCRIPT and STYLE start-tag rules and added 8 prefix names (brx, scripts, stylesheet, style-x, script-x, scriptx, styles, brand), each as a start tag, an end tag and a start tag with an attribute: 24 probe items, 88 in all. The stock main jar fails 3,960 to 5,152 runs already at up to 2 items, depending on the configuration; the fixed scanner fails 0 at up to 3.
• The finder bisected the behaviour to #15756 (10.4.0 passes, 10.5.0 fails), ran a differential of 10.4.0 vs main vs the fixed grammar over all 551,880 strings of up to 4 symbols, and wrote the public-API JUnit and the fix.