HTMLStripCharFilter drops the last word when the text ends right after an unclosed tag name (a < b)

HTMLStripCharFilter outputs an unclosed tag-like construct as text: a < b c strips to a < b c, and Lucene's own testMalformedHTML requires <ahref=http:.. and <a href=" at the end of the input to come out unchanged. But when the input ends right after the tag name, with no character after it, the scanner's end-of-input switch drops the pending <name: a < b strips to a , price a < b is analyzed as [price, a] and a TermQuery for b misses the document, while price a < b (one trailing space) gives [price, a, b]. It has been this way since the JFlex rewrite in 3.6.0 (2012); the filter it replaced kept the text. No exception and no offset error was seen; what is lost is the pending <name at the end of a value or query, usually one word. A one-hunk grammar change fixes it; one upstream test input needs a one-character change.

• Pending segment: when the scanner sees <, it keeps the characters it reads in a buffer until it knows whether they form a tag. A closed tag is removed (or replaced by a newline for block elements); anything else is given back as text.
• Tag-tail state: the scanner state after < (optionally whitespace or /) and a tag name, waiting for attributes and the closing >. The grammar has six, three after < and three after </ (HTMLStripCharFilter.jflex:48-49); four of them matter here: START_TAG_TAIL_EXCLUDE (inline elements such as a, b), START_TAG_TAIL_SUBSTITUTE (any other name, such as p) and the matching END_TAG_TAIL_EXCLUDE and END_TAG_TAIL_SUBSTITUTE after </. The other two, START_TAG_TAIL_INCLUDE and END_TAG_TAIL_INCLUDE, are for names listed in escapedTags.
• Name: [:_\p{ID_Start}] [-.:_\p{ID_Continue}]* (HTMLStripCharFilter.jflex:69), so b. and i-n are one name each, and a digit cannot start one (a < 5 is never a tag).

1. < starts the pending segment (HTMLStripCharFilter.jflex:334-339); whitespace or / may follow (:645-649); a name moves to a tag-tail state (:693-712 for start tags, :591-609 for end tags).
2. In a tag-tail state the only rule is the tail that ends in > (:714-745, :611-641). If more input follows but no > completes the tag, the shared fallback rule (:914-922) pushes the character back and outputs the pending <name unchanged.
3. If the input ends while the scanner is still in a tag-tail state, JFlex runs the %eof block (:275-323). It lists the four tag-tail states in its "Exclude" group (:284-288), which clears the pending segment. The generated scanner has the same switch in zzDoEOF() (HTMLStripCharFilter.java:6133-6185).
4. Name is matched greedily, so a following name character (the . of b.) only makes the name longer. Any other character either completes the tag with a tail ending in > (stripped) or, with no >, triggers the fallback of step 2. So the scanner is still in a tag-tail state at end of input only when the input ends right after the (longest) name: the Exclude grouping changes the output in exactly that case.
5. The states one step earlier (<, </, < ) and the escaped-tag tails are in the "Include" group (:306-318), which outputs the pending segment. So a < and a < are kept, a < b is not, and a < b is.

Kept at end of input: a <, a < , a </, a < 5, a < b c, <ahref=http:.., <a href=", and a name listed in escapedTags (one of the two *_INCLUDE states).

A state-table model parsed from the grammar (22 states) finds exactly 5 states whose end-of-input action differs from what they do when more input follows: these 4, and the character-reference tail, whose end-of-input leniency is commented ("At end of file, allow char refs without semicolons", :298) and tested.

• Who is exposed. Applications that strip HTML with HTMLStripCharFilter: directly, through the htmlStrip SPI factory (for example in a CustomAnalyzer), or through a search server's char filter built on it (Solr's solr.HTMLStripCharFilterFactory; other servers call theirs html_strip; not run on a downstream product). No Lucene analyzer adds it by default, so it is opt-in, but it is stock and needs no non-default setting.
• What happens. The last word of the value is not indexed, so a term query for it misses the document. No error, and no offset error was seen (the final offset was right in every case of the hunt's bounded sweeps).
• What it takes. A field value or query string that ends exactly in < [whitespace] name or </ name with no >: plain-text comparisons (a < b, for i<n, if x<y), or text cut off inside a tag (...<b, where losing the fragment does little harm). By reading (not run), each value of a multi-valued field has its own end, and a query string is hit the same way when it goes through the full analysis chain. Deterministic; index size does not matter, and no non-default setting is needed (a name listed in escapedTags is kept).
• How much is lost. The pending <name at the very end: one JFlex Name, usually one word. A name may contain -, ., :, _ and digits, so a tokenizer can split it: price a < foo-bar gives [price, a], while price a < foo-bar gives [price, a, foo, bar].
• Versions. Reproduced on 21 releases and snapshots from 3.6.0 to 10.5.2 plus main, and on a branch_10_6 build at 4ae946e5a9b (10.6.0); the end-of-input grouping is unchanged in the history between them.
• How likely. Rare: such endings are a small share of real fields, and no report of it was found.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: expected:<a [< b]> but was:<a []>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.analysis.charfilter.TestHTMLStripUnclosedTagAtEnd.testUnclosedTagAtEndOfInputIsKept(TestHTMLStripUnclosedTagAtEnd.java:32)

• Lucene's tests keep unclosed tags with an attribute part at end of input. testMalformedHTML maps <a href=" to <a href=" (TestHTMLStripCharFilter.java:194-195) and <ahref=http:.. to <ahref=http:.. (:218-219): unclosed start tags that run to the end of the input, kept. No test covers a bare <name at the end; it is inconsistent with these recovery cases. testCDATA labels its unclosed <! cases "unclosed angle bang - all input is output" (:502, :508; added by SOLR-5983 in 2014, which fixed another end-of-input bug in the same switch).
• The scanner keeps it in the neighbouring cases. Mid-text the fallback rule outputs the unclosed tag (HTMLStripCharFilter.jflex:914-922); at end of input the Include group outputs a pending <, </ or < (:306-318). Whether the name is dropped depends only on whether the input ends right after it, which is hard to defend for plain text (a committer may still see dropping a truncated <div as useful).
• Documented for non-HTML input (in the Solr Reference Guide, not in Lucene's javadoc). The Solr Reference Guide's HTMLStripCharFilterFactory section: "TIP: The input need not be an HTML document. The filter removes only constructs that look like HTML. If the input doesn't include anything that looks like HTML, the filter won't remove any input." Its example table maps if a<b then print a; to itself; cut after b, the filter returns if a (if x<y gives if x). The same page documents the one intended end-of-input special case, the optional ; of a final entity reference; nothing similar exists for tags.
• History. The Solr 3.5.0 filter that the 2012 rewrite (LUCENE-3690) replaced keeps all these inputs. The LUCENE-3690 thread (41 comments on JIRA; 40 in the GitHub mirror) never discusses tags at end of input.
• No javadoc contract either way. The class only "attempts to strip out HTML constructs".

The intended-behaviour checker built the strongest case for "intended": (a) the %eof block lists every state by hand, and the Exclude/Include split for tag tails reads like "a tag in progress at end of input is markup: strip it unless escaped"; (b) for < directly followed by a letter, browsers agree: the HTML tokenizer's eof-in-tag rule turns a<b into a, and for HTML cut off mid-tag (<p>Hello</p><div) dropping the fragment avoids a junk token; (c) Lucene treats < X > as a tag by design (x < X > gives x \n; LUCENE-5943, Not A Problem), so "< b is not a tag in HTML" is not Lucene's model; (d) upstream testOffsets passes only because the trailing < X is dropped.

Why it fails: the drop depends only on whether the input ends right after the (longest) name, while Lucene's tests keep the tag when an attribute part follows (<ahref=http:.., <a href="). Browsers also drop <b c and <a href=" at end of input, which Lucene and its tests keep, so browser parity is not the filter's model. LUCENE-5943 is about closed tags. And testOffsets depends on the drop only through its helper, which pairs the n-th output X with the n-th input X: its input predates the rewrite (the old filter output it unchanged), and it passes today only because both the X inside < X > and the trailing < X are removed. Argue from Lucene's own consistency, not from browsers.

Option A was applied in a scratch worktree of main a9779ef3c193, the scanner was regenerated with the build's own regenerateHTMLStripCharFilter task, and the tests were run with assertions on and off. Options B and C are alternatives a committer may raise; neither was run.

In the %eof switch of HTMLStripCharFilter.jflex, remove START_TAG_TAIL_EXCLUDE, START_TAG_TAIL_SUBSTITUTE, END_TAG_TAIL_EXCLUDE and END_TAG_TAIL_SUBSTITUTE from the Exclude group and add them to the Include group, then regenerate the scanner (./gradlew :lucene:analysis:common:regenerateHTMLStripCharFilter, which also updates the generated-source checksums).

In TestHTMLStripCharFilter#testOffsets, change the third input from X < &zz >X &# < X > < &l > &g < X to X < &zz >X &# < Y > < &l > &g < X. Its helper finds the n-th X of the input by indexOf and compares it with the corrected offset of the n-th X in the output; the input has four X's (offsets 0, 9, 16, 32). Today the X at 16 (inside the tag < X >) and the trailing X at 32 are both removed, so only 0 and 9 are checked. With the fix, the trailing X is output at its correct offset 32, and the helper pairs it with the stripped X at 16: expected:<16> but was:<32>. With < Y > the helper checks 0, 9 and 32.

--- a/lucene/analysis/common/src/java/org/apache/lucene/analysis/charfilter/HTMLStripCharFilter.jflex
+++ b/lucene/analysis/common/src/java/org/apache/lucene/analysis/charfilter/HTMLStripCharFilter.jflex
@@ -281,11 +281,7 @@
     case STYLE_COMMENT:
     case SINGLE_QUOTED_STRING:
     case DOUBLE_QUOTED_STRING:
-    case END_TAG_TAIL_EXCLUDE:
-    case END_TAG_TAIL_SUBSTITUTE:
-    case START_TAG_TAIL_EXCLUDE:
-    case SERVER_SIDE_INCLUDE:
-    case START_TAG_TAIL_SUBSTITUTE: { // Exclude
+    case SERVER_SIDE_INCLUDE: { // Exclude
       // add (length of input that won't be output) [ - (substitution length) = 0 ]
       cumulativeDiff += (int) (yychar - inputStart);
       // position the correction at (already output length) [ + (substitution length) = 0 ]
@@ -309,6 +305,12 @@
     case NUMERIC_CHARACTER:
     case END_TAG_TAIL_INCLUDE:
     case START_TAG_TAIL_INCLUDE:
+    // An unclosed tag at end of input is output as text, as it is when the input goes on
+    // after the tag name without a closing >
+    case END_TAG_TAIL_EXCLUDE:
+    case END_TAG_TAIL_SUBSTITUTE:
+    case START_TAG_TAIL_EXCLUDE:
+    case START_TAG_TAIL_SUBSTITUTE:
     case LEFT_ANGLE_BRACKET:
     case LEFT_ANGLE_BRACKET_SLASH:
     case LEFT_ANGLE_BRACKET_SPACE: {        // Include
--- a/lucene/analysis/common/src/test/org/apache/lucene/analysis/charfilter/TestHTMLStripCharFilter.java
+++ b/lucene/analysis/common/src/test/org/apache/lucene/analysis/charfilter/TestHTMLStripCharFilter.java
@@ -356,7 +356,9 @@
     doTestOffsets("X & X ( X < > X");
 
     // test backtracking
-    doTestOffsets("X < &zz >X &# < X > < &l > &g < X");
+    // the X inside "< Y >" would be stripped, and doTestOffsets pairs the n-th output X with the
+    // n-th input X
+    doTestOffsets("X < &zz >X &# < Y > < &l > &g < X");
   }
 
   static void assertLegalOffsets(String in) throws Exception {

Trade-offs

• Output changes only for values that end right after a tag name. HTML cut off inside a tag (...<div) now keeps <div as text, as ...<div cl already does today.
• Offsets: the Include branch adds no correction, because all pending input is output; in the hunt's bounded sweep with the fix the final offset was correct in every case, and only the documented character-reference tail still differs from mid-text (that sweep ran the finder's hand-patched scanner, whose grammar change is the same as fix A's except for its comments).
• A test-side alternative (not run): keep the testOffsets input and make the helper check that each output X maps to an X of the input at a strictly increasing offset. That changes the helper for all three inputs.

What the checks validated

• The 3 new tests and the standalone JUnit pass; the charfilter package tests pass 69 of 69 with assertions on and off (stock: 5 fail).
• All of :lucene:analysis:common:test with assertions on: 1,915 tests, 18 skipped, 0 failures.
• :lucene:analysis:common:check -x test passes (generated-source checksums, google-java-format, ecjLint, forbiddenApis; run with the standalone JUnit file moved out, as it is not part of the patch).
• The regenerated scanner changes only in zzDoEOF() (12 changed lines; the DFA tables are identical; 428,378 to 428,514 bytes).
• With the grammar fix and the original testOffsets input, testOffsets fails expected:<16> but was:<32>; the changed input passes on stock main and with the fix (the helper's logic run on both).
• A plain-Java probe on the fixed classes (first run, same grammar change) keeps every input of the table (a < b, <ahref, if a < b., <p>Hello</p><div ends in <div).

In the %eof Exclude branch, output the pending segment when its second character is whitespace (< b, </ b) and keep dropping <b.

Trade-offs

Fixes the plain-text form a < b and keeps today's drop of a truncated <div, which a committer may prefer; the existing tests do not force A over B. It matches browsers for < directly followed by a letter, but keeps the inconsistency the tests contradict (<ahref dropped while <ahref and <ahref=http:.. are kept) and adds a special case to the switch. testOffsets needs the same change, because its trailing < X has a space.

Make the end of input also consume <name ... when a tail was read.

Trade-offs

Contradicts the testMalformedHTML gold values (<ahref=http:.., <a href="), the "all input is output" rule for unclosed <!, and SOLR-2328 (Won't Fix), where the maintainers declined to strip a mid-text unclosed tag; the shared fallback rule would need to know about end of input.

• First signal (a probe, not a formal method). A planning probe on the stock main jar, using only the public API, stripped prefixes of x< notatag more text here and more words and a hand-listed set of inputs, and showed x< not giving x and a < b giving a , while x< notatag m and x < y z were kept.
• State-table model. A model parsed from the real .jflex covered all 22 exclusive lexical states, with one obligation: a state whose mid-text failure gives the pending input back must also output it at end of input. It flagged exactly 5 states: the 4 tag-tail states and the documented character-reference tail.
• Bounded-exhaustive sweep. On the stock jar, with a metamorphic oracle (appending " " or " z " must not change the earlier output): 36,012,942 strings over 18 symbols up to length 6, and 34,636,832 strings over a 32-token grammar up to length 5, each stripped 3 times. Violations occurred in exactly those 5 states and in none of the 16 other reached states, with 0 final-offset errors. With the fix, only the character-reference tail remains.
• The finder then wrote the public-API JUnit, the fix and the version runs; independent checkers followed (see "Independent checks").