HTMLStripCharFilter drops visible text when a quoted attribute value ends with a backslash

HTMLStripCharFilter's quoted-value macros (HTMLStripCharFilter.jflex:110-111) accept a backslash followed by the quote character as part of the value, a rule from the 2012 JFlex rewrite that is also used for server-side-include strings. HTML has no such escape: a quoted value ends at the next matching quote. When a value ends in a backslash, as a Windows folder path does (href="C:\Temp\"), the scanner can read the backslash and the real closing quote as an escaped quote and keep the value open to the next quote of the same kind. The start-tag rule takes the longest match, so that reading wins: everything from the first tag to the end of the next tag that has such a quote is removed as one tag, and the visible text in between never reaches the index. <a href="C:\Temp\">folder</a> text after <a href="x">y</a> end strips to y end instead of folder text after y end; a TermQuery for a hidden word misses the document, CheckIndex is clean and no exception is raised. The 20 event attributes the grammar lists (onclick, onload, ...) have behaved this way since 3.6.0; #11724 (9.9.0) reused the macros for every quoted attribute whose name matches the grammar's Name macro, so href, title, alt, class, data-* and other on* values lose text since 9.9.0 (9.8.0 kept it). The run-on tag can also swallow a <script> start tag, and then script text is indexed. The fix gives attribute values their own macros without the escape; with it, :lucene:analysis:common:test passes (1,913 tests, 18 skipped, 0 failures).

• HTMLStripCharFilter: a CharFilter in lucene/analysis/common that removes HTML markup and decodes character references before tokenization; it is used directly, through the htmlStrip factory, or through downstream html_strip char filters. Its scanner is generated by JFlex from HTMLStripCharFilter.jflex.
• Quoted attribute value: the "..." or '...' after name= in a start tag. In HTML it ends at the next quote of the same kind; only &, NUL and end of input are special inside it (WHATWG 13.2.5.36-37).
• Longest match: JFlex picks the rule and the reading that consume the most input. When the grammar allows two readings of a start tag, the longer one wins.

1. The value macros accept an escaped quote (HTMLStripCharFilter.jflex:110-111):

   SingleQuoted = ( "'" ( "\\'" | [^']* )* "'" )
   DoubleQuoted = ( "\"" ( "\\\"" | [^\"]* )* "\"" )

Inside the quotes: a backslash followed by the quote character, or any run of other characters, <, > and \ included. So "C:\Temp\" matches either as the value C:\Temp\ or with \" as an escaped quote and the value still open.

1. Attribute rules use these macros: EventAttribute (:113, since 3.6.0) for the 20 event attributes listed in EventAttributeSuffixes (:89-109: onclick, onload, onmouseover, ...) and OpenTagAttribute (:114, added by #11724 in 9.9.0) for every name= attribute whose name matches Name (:69), both inside OpenTagContent (:115).
2. The start-tag tail ( "="\s* {OpenTagContent} | {OpenTagContent} )? \s* "/"? ">" (:714-745) takes the longest match. For <a href="C:\Temp\">folder</a> text after <a href="x">y</a> end:
   • reading 1 ends the value at the real quote and the tag at the next >;
   • reading 2 takes \" as escaped, runs through >folder</a> text after <a href= to the next ", eats x" as plain tag text and ends at the second tag's >. Reading 2 is longer and wins: <a href="C:\Temp\">folder</a> text after <a href="x"> is one "tag", removed without replacement (an inline element) or replaced by a newline (a block element, :149).
3. Reading 2 needs a later quote of the same kind ([^"]* skips single quotes) and then a > with no < in between. If a later attribute with the same quote kind follows in the same tag, both readings end at the same > and nothing is lost; a later attribute in the other quote kind does not help: <a href="C:\Temp\" class='c'> still gives y end.
4. Before #11724, ordinary attribute values (and on* names outside the list) were consumed one character at a time by [^<>], so the first > ended the tag: 9.8.0 keeps the text. #11724 added OpenTagAttribute so that < and > inside quoted values no longer end a tag, and reused the macros, escape included.

Also correct on main: a backslash in the middle of the value, no later quote of the same kind, only a later quote of the other kind outside the tag, a later same-kind quote followed by a < before any >, or an attribute name that Name rejects (@click). In a probe where 40 words without a double quote followed the trigger, all 40 were dropped. With escapedTags containing the tag, the run-on start tag is output verbatim instead, the swallowed text and markup included.

• Who is exposed. Users of HTMLStripCharFilter directly, through the htmlStrip SPI factory (HTMLStripCharFilterFactory.java:42), or through a downstream html_strip char filter built on it. No Lucene analyzer adds it by default: opt-in, but a stock component that needs no special setting.
• What the damage looks like. Browser-visible text is missing from the char-filter output and so from the index: a term query for a hidden word misses the document and delete-by-term leaves it live, while text after the hidden span is indexed normally and CheckIndex is clean. No exception, with assertions on or off. When the run-on tag swallows a <script> start tag, the script body is indexed instead: the same bug can add terms.
• What it takes. A quoted attribute value whose last character is a backslash, no later quote of the same kind in the same tag, and a later quote of the same kind in the document followed by a > with no < in between. Plausible sources: Windows paths (href="C:\Temp\", UNC \\server\share\), alt="\" for a backslash key, code or regex samples. HTML escaping leaves a backslash unchanged, so plain text that a site escapes into an attribute can trigger it too. How often this occurs in real pages was not measured.
• How much is lost. Everything up to that later quote and the next >: usually the text up to the next tag with a quoted attribute, a whole paragraph in the block case, 40 words in one probe. Deterministic in every run (inputs up to about 1,000,000 characters).
• Versions. Quoted attributes whose names match Name: 9.9.0 to 10.5.2 and main; the 20 listed event attributes (onclick, ...): from 3.6.0.
• How likely. Rare: values ending in a backslash are uncommon in real pages, and no report was found since 9.9.0 (or 2012 for the listed event attributes).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: expected:<[folder text ]y> but was:<[]y>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.analysis.charfilter.TestHTMLStripAttributeBackslash.testAttributeValueEndingInBackslash(TestHTMLStripAttributeBackslash.java:33)

• HTML rules. The WHATWG tokenizer's attribute value (double-quoted) and (single-quoted) states end the value at the matching quote; only &, NUL and end of input are special. Python's html.parser, run as an independent reference, reads href as C:\Temp\ and keeps folder text after y end, also for title, onclick and the paragraph case.
• Earlier behaviour. 9.8.0, 8.11.4, 4.0.0 and 3.6.0 keep the text for ordinary attributes (executed); the releases in between have the same OpenTagContent line, without OpenTagAttribute.
• #11724's goal. Its PR body: "'<' and '>' characters are acceptable in attribute values per the HTML5 spec". A reviewer (rmuir) looked at the generated code: "I only see smaller DFAs, not any 'logical changes'". Neither the PR nor #11556 mentions backslashes. CHANGES lists it under 9.9.0 Bug Fixes (CHANGES.txt:3278).
• Precedent. "GITHUB#15754: Fix HTMLStripCharFilter to prevent tags from incorrectly consuming subsequent HTML when an attribute value ends with an equals sign." (CHANGES.txt:1160-1161, 10.5.0 Bug Fixes). Its four tests (TestHTMLStripCharFilter.java:652-713) expect only the tags to be removed.
• Where the escape came from. The 2012 grammar introduced the escape-aware macros for server-side-include strings and the listed event attributes at the same time; nothing records them as meant for ordinary attribute values. The upstream tests that use \' or \" do so inside a server-side-include string (testServerSideIncludes, :400-411) and inside a <!-- --> comment in a script (testScriptQuotes, :413-422). Quoted strings in such comments, and in server-side includes nested in comments, have their own states with an escape rule (HTMLStripCharFilter.jflex:848-858). No upstream test has an attribute value ending in a backslash; the one testMalformedHTML input with \" in an attribute (:168) strips the same either way, and every :lucene:analysis:common test passes with the fix.
• No contract for the escape. The class javadoc only says it "attempts to strip out HTML constructs" (HTMLStripCharFilter.jflex:31); nothing documents backslash escapes in attribute values. The contract is behavioural, not a javadoc promise.

The strongest case for "intended": (a) the escape is deliberate grammar, written in the 2012 rewrite and applied to the listed event attributes from the start, so 9.9.0 could be read as a deliberate extension; (b) the filter is best effort ("attempts to strip"), and committers have said that "right" is not well defined for broken HTML; (c) the escape helps some malformed input: for <a title="5 \" > 3">x</a> after, main gives x after, closer to what the author meant, while 9.8.0, the fix and html.parser give 3">x after; (d) the event-attribute half has stood for nearly 15 years (since January 2012) without a report found, and changing it alters indexed output for existing users.

Why it fails: the trigger is valid HTML that 9.8.0 handled correctly, and the best-effort caveat covers malformed input; #11724 aimed at HTML5 attribute semantics, and no recorded discussion in it addresses backslashes; the case (c) benefit is narrow, while the cost falls on valid documents, up to whole paragraphs; for event attributes the HTML tokenizer also ends the attribute at the quote, whatever JavaScript does inside; and a long silence is not a decision: no issue, test or discussion accepts it. The case for "intended" is moderate for the event-attribute half: a committer may prefer to keep it, and fix option B does that.

Three grammar options were applied in a scratch worktree of a9779ef3c193, each with the scanner regenerated by ./gradlew :lucene:analysis:common:regenerateHTMLStripCharFilter (JFlex 1.9.1; it also updates the checksum file), and run against TestHTMLStripCharFilter with the four new tests, assertions on and off.

Add AttributeSingleQuoted / AttributeDoubleQuoted without the \' / \" alternative, use them in EventAttribute and OpenTagAttribute, leave SingleQuoted / DoubleQuoted for server-side-include strings, then regenerate HTMLStripCharFilter.java.

 SingleQuoted = ( "'" ( "\\'" | [^']* )* "'" )
 DoubleQuoted = ( "\"" ( "\\\"" | [^\"]* )* "\"" )
 ServerSideInclude = ( "<!--#" ( [^'\"] | {SingleQuoted} | {DoubleQuoted} )* "-->" )
-EventAttribute = \s+ [oO][nN] {EventAttributeSuffixes} \s* "=" \s* ( {SingleQuoted} | {DoubleQuoted} )
-OpenTagAttribute = \s+ {Name} \s* "=" \s* ( {SingleQuoted} | {DoubleQuoted} )
+// HTML attribute values have no backslash escapes (unlike the server-side-include strings above)
+AttributeSingleQuoted = ( "'" [^']* "'" )
+AttributeDoubleQuoted = ( "\"" [^\"]* "\"" )
+EventAttribute = \s+ [oO][nN] {EventAttributeSuffixes} \s* "=" \s* ( {AttributeSingleQuoted} | {AttributeDoubleQuoted} )
+OpenTagAttribute = \s+ {Name} \s* "=" \s* ( {AttributeSingleQuoted} | {AttributeDoubleQuoted} )
 OpenTagContent = ( {EventAttribute} | [^<>] | {ServerSideInclude} | {OpenTagAttribute} )*

Trade-offs

• Also changes the listed event attributes (onclick, ...), which have read \" as an escape since 2012 (the HTML rule applies to them too).
• A malformed value naively escaped with a backslash and containing > after the escape now strips like 9.8.0 and HTML parsers do, not like main.
• Server-side includes keep the escape, also inside a tag ({ServerSideInclude} in OpenTagContent still uses the escape-aware macros).
• The new tests check the stripped text only, not offsets; the existing offset tests pass.

What the checks validated

• The four new tests pass; TestHTMLStripCharFilter 40 of 40 with assertions on and off.
• All of :lucene:analysis:common:test (assertions on): 339 classes, 1,913 tests, 18 skipped, 0 failures.
• :lucene:analysis:common:check -x test passes (generated-source checksums, google-java-format, ecjLint, forbiddenApis).
• The generated scanner shrinks from 428,378 to 391,691 bytes.
• Earlier, with the hunt's own regeneration: the independent checkers' tests and probes keep the text, and TestHTMLStripCharFilter plus TestHTMLStripCharFilterFactory pass (43 tests).
• With the fix's scanner shadowed, the other-quote-kind cases, onpointerdown and the <script> case keep their text (resolution probe).

Use the escape-free macros only in OpenTagAttribute; keep EventAttribute as it is.

Trade-offs

Keeps the listed event attributes' behaviour of nearly 15 years, bug included: for onclick both rules match and the longer escape reading still wins. The generated scanner grows from 428,378 to 7,164,352 bytes.

What the checks validated

The double- and single-quoted tests and the control pass; testBackslashAtEndOfEventAttributeValue still fails (40 tests, 1 failed, assertions on and off).

Change SingleQuoted / DoubleQuoted themselves to ( "'" [^']* "'" ) / ( "\"" [^\"]* "\"" ): a 2-line diff.

Trade-offs

Smallest diff, but it also removes the escape from server-side-include strings inside a tag, one of the two places the 2012 grammar used it. The one upstream test with escaped quotes in an in-tag server-side-include string (testServerSideIncludes, TestHTMLStripCharFilter.java:403) has balanced escaped quotes and strips the whole tag either way, so it passes; no upstream test depends on the escape there, and the effect on input that does is unverified. A changes less for the same benefit.

What the checks validated

The four new tests and all 40 TestHTMLStripCharFilter tests pass with assertions on and off, testServerSideIncludes included; all of :lucene:analysis:common:test passes (1,913 tests, 18 skipped, 0 failures); scanner 364,122 bytes. check was not run for this option.

Method: random grammar-based fuzzing (registry tier non-formal), a hunt unit.

• The fuzzer builds random HTML-like documents from constructs the HTML5 tokenizer defines (not checked by a conformance validator) and checks the oracle "every plain-text word between constructs appears in the output", then minimises each violation to a shape.
• Stock run (seed 1, 100,000 documents): 8,977 documents lost text, 8,472 distinct minimal shapes; most were script/style cases (the sibling LFV-311), and some had a backslash before a value's closing quote.
• With script/style content and backslashes excluded (seed 2, 300,000 documents), only 12 shapes remained, all with name= inside a value (the sibling LFV-290). Excluding those too (seed 3), 1,000,000 documents lost nothing, which isolated this class.
• The finder then wrote the public-API JUnit, bisected with release jars and a commit-level scanner swap, and wrote the grammar fix.