HTMLStripCharFilter leaves six-hex-digit character references such as é undecoded: its length check counts the 'x'

HTMLStripCharFilter turns numeric character references back into the characters they name before the text is tokenized. Its hexadecimal rule is meant to accept up to six hex digits, enough for the largest code point 10FFFF, as the comment next to the check says ("10FFFF: max 6 hex chars"). But the length it checks is the length of the whole match, x included, so only five digits get through.

A reference with exactly six hex digits, such as the zero-padded é (é) or the plane-16 􀀀 and , is therefore emitted unchanged, as literal text, while é, é and  are decoded. With StandardTokenizer, bad is indexed as b, x000061, d, and a TermQuery for bad misses the document. It has been this way since the JFlex scanner shipped in 3.6.0. The fix is one character in the grammar plus regenerating the scanner.

After &# the scanner enters the state NUMERIC_CHARACTER (HTMLStripCharFilter.jflex:351). Its hex rule (:494-527):

[xX] [0-9A-Fa-f]+ {                       // the match is the x plus every digit
  int matchLength = yylength();             // :495, counts the x
  ...
  if (matchLength <= 6) { // 10FFFF: max 6 hex chars        :497
    String hexCharRef
        = new String(zzBuffer, zzStartRead + 1, matchLength - 1);   // skips the x
    ...
    if (codePoint <= 0x10FFFF) { ... decode ... }            :506
    else { pass through as text }                            :517-521
  } else { pass through as text }                            :522-526

Six hex digits make matchLength 7, so the reference takes the last branch and its original characters are emitted unchanged. The decimal rule (:528-531, [0-9]+) matches digits only and checks matchLength <= 7 // 0x10FFFF = 1114111: max 7 decimal chars, which is right. The generated scanner has the same two checks (HTMLStripCharFilter.java:6557 and :6327).

reference      digits  output
a         2 hex   a              decoded
a      5 hex   a              decoded
&#x000061;     6 hex   &#x000061;     NOT decoded  <- the bug
&#x0000061;    7 hex   &#x0000061;    literal by design, also after the fix
&#0000097;     7 dec   a              decoded
&#00000097;    8 dec   &#00000097;    literal by design

End to end (StandardTokenizer + LowerCaseFilter): bad gives the tokens b (offsets 0-1), x000061 (3-10) and d (11-12); TermQuery(body:bad) finds the 2-digit, 5-digit and 7-decimal-digit documents (3 hits) and misses the six-digit one; CheckIndex is clean. a  z gives a, x10ffff, z.

• Trigger. A hex reference with exactly six hex digits whose value is at most 0x10FFFF: a plane-16 character (U+100000 to U+10FFFD, Supplementary Private Use Area-B; U+10FFFE and U+10FFFF are noncharacters) or a smaller code point zero-padded to six digits (HTML allows leading zeros). Reached through new HTMLStripCharFilter(reader) or the factory (SPI name htmlStrip), with no option to set.
• Effect. The reference is indexed as text, for example the token x000061 or x10ffff, so a word that contains it is not found by its decoded form. The text around it is tokenized normally, offsets point at the original text, nothing throws, CheckIndex is clean.
• Not affected. Every code point below U+100000 in its shortest hex form (at most five digits), decimal references of up to seven digits, and named entities.
• Likelihood. Rare: the recorded searches found no public report of it since it shipped in 2012.

new HTMLStripCharFilter(new StringReader("é")) is read to the end with Reader.transferTo and the output is compared with é. Deterministic, single thread; LuceneTestCase is only the base class. é is an ordinary BMP character, so the expected value needs no surrogate pair; plane-16 references such as 􀀀 take the same branch.

Regression test: this same class (or its one method added to TestHTMLStripCharFilter); it passes with the recommended fix.

cp $REPRO/TestHTMLStripHexCharRefSixDigits.java $LUCENE/lucene/analysis/common/src/test/org/apache/lucene/analysis/charfilter/
cd $LUCENE && ./gradlew :lucene:analysis:common:test --tests TestHTMLStripHexCharRefSixDigits -Ptests.seed=DEADBEEF -Ptests.asserts=true
# assertions off fails the same way: -Ptests.asserts=false

# gradle, stock 645d4fe14a20, -Ptests.seed=DEADBEEF -Ptests.asserts=true (stack trimmed; -Ptests.asserts=false identical)
   >     org.junit.ComparisonFailure: expected:<[é]> but was:<[&#x0000E9;]>
   >         at __randomizedtesting.SeedInfo.seed([DEADBEEF:54CBF2DB236B7CEA]:0)
   >         at app/junit@4.13.2/org.junit.Assert.assertEquals(Assert.java:117)
   >         at app/junit@4.13.2/org.junit.Assert.assertEquals(Assert.java:146)
   >         at app//org.apache.lucene.analysis.charfilter.TestHTMLStripHexCharRefSixDigits.testSixHexDigitReference(TestHTMLStripHexCharRefSixDigits.java:32)

:lucene:analysis:common:test (FAILURE): 1 test, 1  failure

# standalone runner, seed DEADBEEF: main 645d4fe14a20 (-ea and -da) and 10.6.0 (-da) fail with the same message

• The stated limit. HTMLStripCharFilter.jflex:497: "10FFFF: max 6 hex chars": six hex digits, the length of the largest code point. The implementation admits five.
• The code knows the x is in the match. :498-499 parses new String(zzBuffer, zzStartRead + 1, matchLength - 1).
• The sibling is right. :531 admits seven decimal digits for 1114111;  and é decode,  and é do not (executed).
• Dead code. With at most five digits the largest value is 0xFFFFF, so the codePoint <= 0x10FFFF else-branch (:517-521) is unreachable.
• No checked document states five digits. Not the class javadoc (:31), the factory or the package docs, no CHANGES or MIGRATE entry, and not the LUCENE-3690 discussion (#4764). Upstream tests assert that numeric references decode (TestHTMLStripCharFilter.java:115-119 testEntities, :601-634 testUTF16Surrogates) but never use more than four hex digits.
• Partly a regression, in lineage only. The Solr class that LUCENE-3690 replaced read up to 10 hex digits and decoded é (executed on the Solr 3.5.0 jar). It is not a regression between Lucene releases: Lucene 3.5.0 had no Lucene HTMLStripCharFilter. Neither version ever decoded six-digit supplementary references correctly.

The strongest case for "intended": the class only "attempts to strip out HTML constructs" (HTMLStripCharFilter.jflex:31); it does not follow HTML5's numeric-reference rules anyway (values above 0x10FFFF and long zero padding stay literal), so a digit cap is an implementation choice; and six-digit references are uncommon. A cross-model reviewer rated this "moderate".

Why it fails: the cap's intended value is written next to the check, and the code misses its own stated limit by one. That is an implementation slip, not a choice about HTML5 conformance, and "uncommon" is about impact, not intent. Every reviewer and the adjudication rejected the argument.

One fix was written and validated in a scratch worktree of 645d4fe14a20; the alternatives are described but not run.

Change the bound in the grammar and run ./gradlew :lucene:analysis:common:regenerateHTMLStripCharFilter, which rewrites HTMLStripCharFilter.java:6557 the same way (the only change in the generated file) and updates src/generated/checksums/regenerateHTMLStripCharFilter.json: 3 files, +4/-4. The regenerated scanner is byte-identical to the finder's hand-edited copy.

   [xX] [0-9A-Fa-f]+ {
     int matchLength = yylength();
     inputSegment.write(zzBuffer, zzStartRead, matchLength);
-    if (matchLength <= 6) { // 10FFFF: max 6 hex chars
+    if (matchLength <= 7) { // "x" + 10FFFF: max 6 hex chars
       String hexCharRef
           = new String(zzBuffer, zzStartRead + 1, matchLength - 1);

Trade-offs

• A visible analysis change for every hex reference with exactly six digits, not only the headline examples. Each now decodes the way its shorter spelling already does (executed): � becomes U+0000 like �; the zero-padded surrogate � becomes U+FFFD like � (HTMLStripCharFilter.jflex:509-511; the paired-surrogate rules at :355-489 match only adjacent unpadded pairs); an unterminated &#x000061 at the end of the input becomes a like &#x00061.
• The tokens of such text change, so documents indexed before the fix keep the old tokens until they are reindexed. Worth a CHANGES entry under Bug Fixes.
• Unchanged: seven or more hex digits and eight or more decimal digits stay literal (the stated limits); values above 0x10FFFF such as � stay literal through the range check.

What the checks validated

Gradle --no-daemon in a scratch worktree of 645d4fe14a20, OpenJDK 25.0.3 (2026-10-31). Stock: the repro fails with assertions on and off. With the fix:

• the repro passes with assertions on and off;
• org.apache.lucene.analysis.charfilter.*: 64 tests (TestHTMLStripCharFilter 36, TestHTMLStripCharFilterFactory 7, TestMappingCharFilter 18, its factory 2, the repro 1), 0 failures, seeds DEADBEEF and CAFEBABE;
• the whole :lucene:analysis:common:test, seed DEADBEEF: 340 test classes, 1,910 tests, 18 skipped, 0 failures (counted from the test-result XML; an independent fact check's own run got the same totals);
• regenerateHTMLStripCharFilterChecksumCheck passes;
• end to end (the same change on the earlier main): bad becomes the token bad and TermQuery(body:bad) finds it (4 hits instead of 3); an independent 7-test check also passes, including � staying literal.

int hexDigits = matchLength - 1; if (hexDigits <= 6), and use hexDigits in the substring.

Trade-offs

Same behaviour as the recommended fix and reads more clearly; a slightly larger diff.

Skip leading zeros, then cap at six significant hex digits (and seven decimal), as HTML5 decoders do.

Trade-offs

Also decodes a, but changes the stated limits for seven or more digits: a behaviour change beyond this bug. Dropping the cap alone is not an option: Integer.parseInt throws for values of 0x80000000 and above, and its catch block only asserts (HTMLStripCharFilter.jflex:501-505): with assertions off codePoint stays 0 and the reference would become U+0000.

Method: non-formal (code reading, then an executed test).

• Finder. The hunt unit for analysis char filters and stemmers was building a grammar fuzzer for HTMLStripCharFilter whose fragment list includes  and . Reading the hex rule showed the check counts the x. The fuzzer's oracles (offset range and order, final offset, chunking independence) reported 0 violations over 2,000,000 documents, because an undecoded reference breaks none of them.
• Confirmation. A public-API JUnit (3 six-digit cases fail, the five-digit and decimal controls pass) on main with assertions on and off and on 9.12.3, and a main() probe on 9 releases back to 4.0.0.
• Hunt. Independent reproduction, cross-model reviews, an adjudication, the version-history runs, a prior-art search, a security screen and a one-test minimal repro followed (see the checks below).