HTMLStripCharFilter drops all text after a <!-- -->-wrapped script or style block with an apostrophe in a JavaScript or CSS comment

An old HTML idiom wraps the code of a <script> or <style> block in <!-- ... --> (upstream's own test fixture htmlStripReaderTest.html does). Inside such a section HTMLStripCharFilter tracks JavaScript/CSS string literals, so that a quoted --> or </script> cannot end the block, but it knows nothing about JavaScript or CSS comments or line ends. An apostrophe in // Don't remove or /* Outlook's spacing fix */, or a lone ", therefore opens a "string" that runs past the closing --> and the end tag. A string state consumes everything after the quote without output; if no matching quote follows, the input ends in that state and the %eof handler just ends the stream, so the visible text after the correctly closed block never reaches the tokenizer, the field is indexed without it, and nothing reports an error. In the executed controls, the same block without the quote, without the wrapper, or with balanced quotes keeps the text. The rules date from the JFlex rewrite of the filter (LUCENE-3690, 2012) and are unchanged; every release from 3.6.0 to 10.6.0 (10.6.0 by reading) and main is affected. The filter it replaced kept this text: it rolled back when it could not match a tag and its end tag.

• HTMLStripCharFilter: a CharFilter in analysis-common that removes HTML markup before tokenization. It is generated by JFlex from HTMLStripCharFilter.jflex; the scanner moves between named lexical states.
• The <!-- --> wrapper: an old idiom that hides script or style code from very old browsers: <script><!-- ... //--></script>, <style><!-- ... --></style>. It is valid HTML and browsers show the text after the block.

Line numbers are HTMLStripCharFilter.jflex at a9779ef3c193.

1. <script ...> enters SCRIPT. Script content is dropped and the end tag becomes one newline (SCRIPT_REPLACEMENT, :153; end-tag rule :862).
2. <!-- inside SCRIPT enters SCRIPT_COMMENT (:860-861; for style: :887-888 enters STYLE_COMMENT).
3. In SCRIPT_COMMENT (:832-838) ' enters SINGLE_QUOTED_STRING (:834) and " enters DOUBLE_QUOTED_STRING (:835). The only direct route back to SCRIPT is an unquoted --> (:836; <!--# enters a server-side include, :833, that returns to SCRIPT_COMMENT at its own -->, :818); there is no rule for // or /* */ comments and none for </script>. STYLE_COMMENT is the same (:840-846).
4. SINGLE_QUOTED_STRING (:848-852) ends only at the next ' (:850); a backslash escapes one character (:849) and every other character, including line breaks, -->, </script>, <p> and the text, is consumed by [^] { } (:851). DOUBLE_QUOTED_STRING is the same for " (:854-858).
5. With no second apostrophe the input ends inside the string. Nothing after the apostrophe was output: the string state consumed it with [^] { }. %eof (:275-296) lists SCRIPT_COMMENT, STYLE_COMMENT, SINGLE_QUOTED_STRING and DOUBLE_QUOTED_STRING (:279-283) in the branch marked // Exclude (:288), which only corrects the offsets, clears outputSegment (:293) and ends the stream (:294). No fallback re-reads the consumed input as text, so everything from the apostrophe on is lost.

A later quote, such as Tom's in the body text, ends the "string" there, but the scanner then still needs an unquoted --> and an end tag, so the text after it is usually lost too: <script><!--\n// it's fine\n//--></script><p>Tom's book</p><p>after</p> strips to "".

• The same block keeps the text without the quote, without the <!-- --> wrapper (outside a <!-- section SCRIPT and STYLE do not track quotes), or with balanced quotes (var s = "Don't";). Text before the block survives (<p>before</p> stays).
• Any literal <!-- in script or style text opens the quote-tracking section, wrapper or not: an unwrapped inline script var author = 'Bob<!--'; also drops the paragraph after it (the sub-trigger below).
• A weaker sub-trigger, best presented as an open question: an unterminated <!-- inside a script, <p>before</p><script>a<!--</script><p>after</p>, strips to "\nbefore\n", because SCRIPT_COMMENT has no end-tag rule. The pre-3.6 filter skipped end tags inside script comments on purpose (its code comment cites <script><!-- </script> -->foo</script>), so a committer may call this one intended.

• Entry points. new HTMLStripCharFilter(reader) in an analyzer's initReader, the htmlStrip char filter factory (HTMLStripCharFilterFactory.NAME, for example in CustomAnalyzer), and downstream wrappers of the filter (by code reading, nothing run: Solr's HTMLStripCharFilterFactory and HTMLStripFieldUpdateProcessorFactory, the html_strip char filter and ingest processor of Elasticsearch and OpenSearch).
• Default vs opt-in. No Lucene analyzer adds the filter, so using it is a choice; the bug then needs no non-default setting of the filter.
• Inputs. Legacy-style HTML: the <!-- --> wrapper inside script or style blocks is an old idiom (upstream's own fixture htmlStripReaderTest.html and testStyle use it; how common it is in current corpora was not surveyed), and the trigger is an ordinary apostrophe or inch sign in a comment. One inserted line // don't cache this value in that fixture cuts the stripped output from 637 words (4,869 characters) to 15 (246 characters).
• Scale and scope. Any index size; per field value. The loss runs from the quote to the end of the value, unless a later quote, an unquoted --> and an end tag let the scanner resynchronise. Other values, fields and documents are unaffected, CheckIndex is clean, offsets stay valid, and if the application stores the original HTML, the stored value still shows text that search cannot find (one that stores the stripped value loses it there too).
• Likelihood: rare. The combination is uncommon in current HTML and the prior-report searches found no report since 2012, but a silent loss like this can go unnoticed in large crawled or e-mail corpora.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected the text after </script> to be kept, actual output: []
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.assertTrue(Assert.java:42)
       org.apache.lucene.analysis.charfilter.TestHTMLStripQuoteInScriptComment.testApostropheInWrappedScriptComment(TestHTMLStripQuoteInScriptComment.java:32)

• Contract. Class javadoc (HTMLStripCharFilter.jflex:31): "A CharFilter that wraps another Reader and attempts to strip out HTML constructs." Text after a correctly closed </script> or </style> is document text, not a construct.
• Upstream tests. TestHTMLStripCharFilter.testStyle (:431-440) strips one<style type="text/css">\n<!--\n@import url('...');\n-->\n</style>two to one\ntwo; testScriptQuotes case 1 (:414-417) strips a wrapped script with quoted <!-- comment --> and \"-->\" to one\ntwo. Every test of the quote tracking uses balanced quotes; we found no test, code comment, CHANGES or MIGRATE entry that accepts losing the rest of the document after a closed element.
• Design intent. In LUCENE-3690 (GitHub #4764), which introduced the JFlex filter, its author (@sarowe) stated the goal "get as much content from as varied a range of sources as possible, and never ever allow the input to bork processing" and called its bad-HTML handling "a superset" of the old filter's. The Solr 3.6.0 CHANGES list of behaviour changes for it does not mention dropping text after a script or style block.
• The predecessor kept the text. The hand-written Solr filter it replaced (kept as LegacyHTMLStripCharFilter) also tracked quotes in script comments, but its look-ahead was bounded, and when it could not match a tag and its end tag it rolled back and emitted the input as text (for an unterminated string in script code its findEndTag notes "what to do about a non-match (non-terminated string?) play it safe and index the rest of the data I guess..."). Executed on the two finder inputs it keeps "Welcome to our store" and "Your order has shipped." (with a stray <), while 4.10.4 and main return "".
• Precedent. GITHUB#15754 (10.5.0, "prevent tags from incorrectly consuming subsequent HTML when an attribute value ends with an equals sign") fixed the same symptom, a scanner state swallowing the text after it, as a bug. GITHUB#11556 (9.9.0) is not a second example: there < or > in an attribute value stopped tag recognition and leaked markup into the output, and the text after the tag survived (executed on the 9.8.0 jars). The loss #15754 fixed first appeared in 9.9.0, with the #11556 fix (an input of that shape keeps its text on 9.8.0 and loses part of it on 9.9.0).
• Separable. The recommended fix keeps both testScriptQuotes cases, testComment and the rest of analysis-common passing (0 failures, 18 skipped), so the deliberate quote tracking and this loss can be told apart.

The strongest case for "intended": the filter is best effort ("attempts to"); quote tracking inside script comments is deliberate and tested (testScriptQuotes case 2, :419-421, keeps a quoted </script> inside a script comment from ending the script, so upstream knowingly removes some text a browser would show); and dropping input at end of input in these states is explicit in %eof (its Exclude branch also covers SCRIPT, STYLE, COMMENT and the tag-tail states, :277-288, so this is a general policy), with testComment (:321-334) asserting that an unterminated <!-- strips to "".

Why it fails: best effort covers ambiguous or broken markup, while this input is valid HTML with every construct closed; the scanner just misreads where the comment ends. The quote tracking exists to keep a quoted --> or </script> from ending the block, and all its tests use balanced quotes. The end-of-input exclusion is right for genuinely unterminated constructs, which is all testComment asserts. The recommended fix shows the two are separable.

Every option except C was applied to a scratch worktree at a9779ef3c193, the scanner regenerated with ./gradlew :lucene:analysis:common:regenerateHTMLStripCharFilter (which also updates src/generated/checksums/regenerateHTMLStripCharFilter.json), and run through gradle with -Ptests.asserts=true and =false; the regenerated scanners were also shadowed into the stock jars to run the hunt's repros and probes. C and D were run by the hunt at ea57360912ed, where the grammar is the same. Options A and B were re-run on 2026-10-27 after a backslash-CRLF rule was added to their string states (the first revision lacked it); the earlier revision is kept. Every command is in the fix folder's commands.txt.

optionA-comment-states.diff: HTMLStripCharFilter.jflex +41 lines and one new TestHTMLStripCharFilter method. In SCRIPT_COMMENT, // enters a new SCRIPT_LINE_COMMENT state that ends at a line break (back to SCRIPT_COMMENT) or at --> (to SCRIPT, so the usual //--> still closes). In SCRIPT_COMMENT and STYLE_COMMENT, /* enters a new block-comment state that ends at */ or, as a safety net for an unterminated /*, at -->. In the two quoted-string states, a CR or LF ends a string that was opened from a script or style comment section: neither JavaScript nor CSS strings can contain an unescaped line break. A backslash before LF or CR is still consumed by the existing escape rule, and a new rule consumes a backslash before CRLF (without it the escape rule would take only the CR, and the LF would end the string early and expose a quoted --> or </script>); strings opened from server-side includes are unchanged. The three new states join the %eof Exclude list; like the existing states they use fixed-length rules (at most three characters) with a one-character fallback, so the scan stays linear.

<SCRIPT_COMMENT> {          // STYLE_COMMENT: the same without "//"
  ...                       // the existing quote and "-->" rules
  "//"    { yybegin(SCRIPT_LINE_COMMENT); }
  "/*"    { yybegin(SCRIPT_BLOCK_COMMENT); }
}
<SCRIPT_LINE_COMMENT>  { [\r\n] { yybegin(SCRIPT_COMMENT); }  "-->" { yybegin(SCRIPT); }  [^] { } }
<SCRIPT_BLOCK_COMMENT> { "*/"   { yybegin(SCRIPT_COMMENT); }  "-->" { yybegin(SCRIPT); }  [^] { } }
<SINGLE_QUOTED_STRING> {    // and DOUBLE_QUOTED_STRING
  "\\" [^] { }               // existing escape rule
  "\\" "\r\n" { }            // new: a backslash before CRLF continues the string
  ...
  [\r\n] { if (restoreState == SCRIPT_COMMENT || restoreState == STYLE_COMMENT) {
             yybegin(restoreState); restoreState = previousRestoreState; } }
  [^] { }
}

Trade-offs

It is still not a JavaScript lexer: a regex literal or a division followed by * can be read as a comment start (it then ends at */ or -->), and template literals (backticks) are not tracked. An unbalanced quote in code, not a comment, on the same line as --> and the end tag, in a value with no later line break, still loses the rest as today (<script><!-- var a = "6 wide; --></script><p>after</p> gives ""). Only CR and LF end a string; U+2028, U+2029 and the CSS form feed do not (not tested). It does not cover the unterminated-<!-- sub-trigger (option B).

What the checks validated

Gradle, assertions on and off: the repro 4 of 4; the charfilter package (TestHTMLStripCharFilter 37 with the new method and both testScriptQuotes cases, Factory 7, the mapping suites 20); all of :lucene:analysis:common:test (340 suites, 1,914 tests, 18 skipped, 0 failures); :lucene:analysis:common:check -x test (checksum checks, google-java-format, ecjLint, forbiddenApis). Shadowed into the stock jars: the finder's JUnit 4 of 4; the verifier's file 9 of 10 (only the unterminated-<!-- test fails); the hunt's 8-test reach probe 7 of 8, including its linear-time test on a 100M-character stream at -Xmx128m (the remaining failure is a one-line page with 'Bob<!--' in an unwrapped inline script string, the sub-trigger again); an unterminated /* repeated over 12.5k-100k characters takes 1.3-4.3 ms with no growth (stock 1.4-3.9 ms); the mutated upstream fixture strips to the same 637 words as the original; a backslash before LF or CRLF inside a string keeps a quoted --> and </script> protected, as on stock (CRLF probe; the first revision leaked ';\r\n--> there). All re-run on 2026-10-27 with the backslash-CRLF rule.

optionAB-comment-states-plus-end-tag.diff: A plus 10 grammar lines and one test method. In SCRIPT_COMMENT and the new script comment states, "</" \s* [sS][cC][rR][iI][pP][tT] \s* ">" { yypushback(yylength()); yybegin(SCRIPT); } (the same for style), so the existing SCRIPT/STYLE rule matches the end tag again. It fixes <script>a<!--</script><p>after</p>.

Trade-offs

It reverses the predecessor's deliberate rule that a </script> inside a script comment is not an end tag: <script><!-- </script> -->foo</script>bar gives "\nbar" today and "\n -->foo\nbar" with B (the first </script> then ends the element; no browser comparison was run). No upstream test pins that case, but a committer may prefer to keep it. B is not fully HTML5-style: a quoted </script> inside the comment section still does not end the element (only option C does that).

What the checks validated

Re-run with A's backslash-CRLF rule. Gradle, assertions on and off: all of analysis-common (1,915 tests, 18 skipped, 0 failures); check -x test passes. Shadowed: the verifier's file 10 of 10; the reach probe still 7 of 8 (in the one-line 'Bob<!--' page the template's closing quote opens a string before </script>); cost 1.4-4.4 ms, no growth.

Delete the two "<!--" rules in SCRIPT and STYLE, so script and style content is never treated as a comment section.

Trade-offs

Fixes every trigger seen, but reverses a tested decision: testScriptQuotes case 2 (a quoted </script> inside a script comment) then outputs hello\n'); -->\n instead of hello\n. Worth proposing only if committers want HTML5 behaviour.

What the checks validated

By the hunt at ea57360912ed (same grammar), not re-run here: the verifier's file 10 of 10, the reach probe 8 of 8, linear on the probe's inputs; TestHTMLStripCharFilter 36 tests, 1 failure (testScriptQuotes).

The hunt's first sketch skipped /* ... */ with one look-ahead rule and ended strings at line breaks.

Trade-offs

Quadratic on an unterminated /*: 6.8 s at 100,031 characters against 0.8 ms on stock, and out of memory on a 100M-character stream at -Xmx128m. It also misses the one-line <script><!-- // don't --></script>. Option A replaces it with explicit states.

What the checks validated

By the hunt at ea57360912ed: the 43 TestHTMLStripCharFilter and Factory tests pass, the verifier's file 9 of 10; the cost probe shows the quadratic growth (ratio about 4 per doubling).

Method: code reading during a fuzzing hunt (a hunt unit, analysis char filters, 2026-10-26, main ea57360912ed). No model checker, SMT solver or prover was involved.

• An AI tool (the finder) read the SCRIPT, STYLE and comment states of HTMLStripCharFilter.jflex while building HtmlLossFuzz, a fixed-seed fuzzer that generates random documents well formed under the HTML5 tokenizer rules, with script and style raw text over an alphabet of quotes, <!--, -->, // and /*. Its oracle: every plain-text word between constructs must appear in the output; each loss is minimised to one construct.
• On the stock filter, raw text with a quote or an open <!-- was the dominant loss class (8,977 of 100,000 documents lost text across all loss classes, seed 1, including other HTMLStripCharFilter bugs). With quote-free, closed <!-- --> sections and the attribute triggers excluded, 1,000,000 documents lost nothing.
• The finder then wrote the public-API JUnit, which fails 3 of 4 on the stock jar with -ea and -da.