UAX29URLEmailTokenizer splits an ftp URL that has no path: ftp://host is not one URL token, http://host is

UAX29URLEmailTokenizer (also used by UAX29URLEmailAnalyzer and the uax29UrlEmail factory) keeps URLs and e-mail addresses whole and says it tokenizes them "according to the relevant RFCs"; its grammar cites RFC 1738 and RFC 3986. RFC 1738 says of the ftp path: "The whole url-path may be omitted, including the "/" delimiting it". The grammar's http rule makes the path optional (UAX29URLEmailTokenizerImpl.jflex:167), but its ftp rule requires a path that starts with / (:177, :180). So http://localhost and ftp://ftp.example.com/ are one <URL> token each, while ftp://ftp.example.com becomes ftp <ALPHANUM> + ftp.example.com <URL>, ftp://localhost two words, ftp://192.0.2.10 ftp + a <NUM>, and ftp://anonymous@ftp.example.org ftp + an <EMAIL> token //anonymous@ftp.example.org, with no error. An exact term lookup of such a URL finds nothing and <URL>-token extraction misses it; a phrase query analyzed the same way still matches. The rule dates from the 2010 URL grammar (LUCENE-2167) and every release from 3.1.0 to 10.6.0 (10.6.0 by reading) and main has it. Making the path optional (({FTPorFILEpath} {FTPtype}?)?) and regenerating the scanner fixes it; the change alters tokens of path-less ftp URLs, so affected fields need a reindex.

• UAX29URLEmailTokenizer: Unicode word breaking (UAX#29) plus URL and e-mail recognition, generated by JFlex from UAX29URLEmailTokenizerImpl.jflex. Used by UAX29URLEmailAnalyzer and the uax29UrlEmail factory. Not the default: IndexWriterConfig defaults to StandardAnalyzer, whose tokenizer has no <URL> type.
• Path: the part after the host, starting with /. RFC 1738 makes it optional for ftp (section 3.2.2) and required for file URLs (fileurl = "file://" [ host | "localhost" ] "/" fpath).

jflex:162  URIauthorityLoose = {URIlogin}? {URIhostLoose} {URIport}?
jflex:167  HTTPurlFull = {HTTPscheme} {URIlogin}? {URIhostLoose} {URIport}? {HTTPpath}? {URIquery}? {URIfragment}?
jflex:177  FTPorFILEpath = "/" {FTPorFILEsegment} ("/" {FTPorFILEsegment})*
jflex:180  FTPurl = {FTPscheme} {URIauthorityLoose} {FTPorFILEpath} {FTPtype}? {URIfragment}?
jflex:183  FILEurl = {FILEscheme} {URIhostLoose}? {FTPorFILEpath} {URIfragment}?
jflex:185  URL = {HTTPurl} | {FTPurl} | {FILEurl}

The http path is optional ({HTTPpath}?); the ftp path is not. The ftp rule reuses the file rule's path production, which is right for file URLs only. Even the ftp fragment can only follow a path.

When FTPurl cannot match, the scanner takes the longest match of its other rules, piece by piece: ftp is an ordinary word (<ALPHANUM>) and :// gives no token. What follows is typed by whichever rule matches it. A host that is a domain with a listed TLD or an IP address ({URIhostStrict}, jflex:157) followed by a port or a fragment is a scheme-less URL, {HTTPurlNoScheme} (jflex:168-174, returned as <URL> at :273), e.g. mirror.example.org:2121. Such a host on its own is a <URL> when a non-word character follows it (jflex:276), and at the end of the text when it is a domain (:283-290) or a bracketed IPv6 address ([::1]). Otherwise it is a word (localhost) or a number (an IPv4 address at the end of the text). With a user name and no password the e-mail rule wins (jflex:297): / is a legal e-mail local-part character (jflex:188), so //anonymous@ftp.example.org is an <EMAIL>; with a password, user:secret is a word.

• Who. Default settings and the public API are enough; no special configuration is needed. It is opt-in: the default StandardAnalyzer has no URL rules. Downstream servers expose the tokenizer too: Solr through the factory (its Reference Guide lists "file://, http(s)://, and ftp:// URLs" among the forms kept as single tokens), and Elasticsearch's uax_url_email tokenizer (its maintainers sent a report about it to Lucene: "this tokenizer belongs to Lucene", quoted in #9092).
• What is lost. One document mirror at ftp://ftp.example.org today, indexed with UAX29URLEmailAnalyzer: TermQuery(body:ftp://ftp.example.org) finds 0 of 1 (1 of 1 for the same text with a trailing /, and 1 of 1 with the fix). Anything that keeps <URL> tokens (link extraction, URL facets, a type filter) gets the bare host or nothing (ftp://localhost, ftp://192.0.2.10 at the end of the text); an e-mail extractor collects //anonymous@ftp.example.org as an address.
• What still works. A query analyzed by the same analyzer is split the same way: the phrase query for ftp://ftp.example.org built by QueryBuilder finds 1 of 1. Word search on the host matches.
• Long URLs. Every tested input is short. As for any token of this tokenizer, a URL longer than the maximum token length (default 255, StandardAnalyzer.java:37; setMaxTokenLength) never comes out as one token, http or ftp, with or without the fix: the javadoc says such tokens "will be chopped up at this token length and emitted as multiple tokens" (UAX29URLEmailTokenizer.java:91-95). No long URL was run.
• How likely. Ftp links are uncommon in today's text and most carry a path (all 154 ftp URLs in each of the two upstream URL fixtures do); nobody has reported it since 3.1.0 (2011). Registry: likelihood rare, priority low.
• Workaround. Write ftp links with a trailing /.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: term 0 expected:<ftp[://localhost]> but was:<ftp[]>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:262)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:513)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:598)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:622)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:694)
       org.apache.lucene.analysis.email.TestFtpUrlWithoutPath.testFtpUrlWithoutPath(TestFtpUrlWithoutPath.java:29)

• Contract. Class javadoc (UAX29URLEmailTokenizer.java:31-33), package javadoc (package-info.java:21-25) and the 3.1.0 CHANGES entry for LUCENE-2167 (CHANGES.txt:17763-17774): URLs and e-mail addresses are tokenized "according to the relevant RFCs". The grammar header lists RFC 1738 and RFC 3986 (jflex:113-122).
• The RFCs. RFC 1738 section 3.1: "/<url-path>" "may be excluded"; section 3.2.2: "The whole url-path may be omitted, including the "/" delimiting it"; section 5: ftpurl = "ftp://" login [ "/" fpath [ ";type=" ftptype ]]. RFC 3986 section 3.3: with an authority, "the path component must either be empty or begin with a slash".
• The author's own description (java-user@): in June 2010, of the NewStandardTokenizer in the then-current LUCENE-2167 patch, whose URL grammar was later moved into UAX29URLEmailTokenizer: "It keeps (HTTP(S), FTP, and FILE) URLs together as single tokens, and e-mails too, in accordance with the relevant IETF RFCs"; in 2012, of UAX29URLEmailAnalyzer: "Full support for URLs with the file:, ftp:, and http/s: protocols".
• Tests. No test pins either behaviour: all 154 ftp URLs in each URL fixture have a path. Deliberate deviations are written down (jflex:172, jflex:275 LUCENE-5391, the mailto: TODO in TestUAX29URLEmailTokenizer); this one is not.
• History. The rule is in the commit that wrote the URL grammar (3c26a9167cc, LUCENE-2167, 2010) and never changed. The issue mentions ftp once: "there is no testing yet for the file:, https:, or ftp: schemes". The author typed both example.com/ and example.com as <URL> because "they effectively refer to the same resource (when interpreted in the context of the HTTP URI scheme)"; they said it of http only, and applying it to ftp://host and ftp://host/ is our reading, which RFC 1738 section 3.2.2 supports. The history shows that no reason for the mandatory ftp path was recorded, not that it was an accident.
• Precedent. LUCENE-8278 (7.4.0 Bug Fixes, CHANGES.txt:7485), LUCENE-5391 (CHANGES.txt:12858) and GITHUB#12561 (11.0.0 Bug Fixes, CHANGES.txt:307) fixed URL and e-mail mis-typing in this grammar as bugs.

The strongest "intended" case (moderate per the cross-model reviews, moderate-to-weak per the independent reproduction; the intended-behaviour check stated it without a rating and rejected it): the tokenizer is a heuristic recognizer, selective elsewhere (four schemes, TLD-gated scheme-less URLs, mailto: dropped); "according to the relevant RFCs" can be read as soundness, not completeness; the author wanted to "only accept strictly correct URLs" (answering a concern about false positives from URL rules and URLs without http://); the rule is 16 years old and every fixture has a path; the host often survives as a <URL> token.

Why it fails: ftp is one of the four supported schemes and the cited RFC says its path may be omitted, so ftp://host is RFC-valid; the "strictly correct" remark is consistent with the fix, though the thread does not decide this case. The author followed each scheme's RFC 1738 BNF elsewhere (optional http path, required file path). The output is not a clean refusal but wrong tokens and types (<NUM>, <EMAIL>), so even the soundness reading is not met. Age and fixtures are not intent (they show only that no reason for the restriction was recorded), and compatibility goes to severity. Verdict: a defect (adjudication: confirmed, high).

One option was built and validated with gradle; the others are noted with their costs and were not run.

One grammar line (jflex:180), the same change as the finder's; ;type= stays tied to a path, as in RFC 1738. Then ./gradlew :lucene:analysis:common:regenerateUAX29URLEmailTokenizer -x :lucene:analysis:common:regenerateTlds rewrites UAX29URLEmailTokenizerImpl.java and its checksum file.

 FTPscheme = [fF][tT][pP] "://"
-FTPurl = {FTPscheme} {URIauthorityLoose} {FTPorFILEpath} {FTPtype}? {URIfragment}?
+FTPurl = {FTPscheme} {URIauthorityLoose} ({FTPorFILEpath} {FTPtype}?)? {URIfragment}?

Trade-offs

• Changes the tokens of every path-less ftp URL: a CHANGES entry with a reindex note; main is the natural target.
• The regenerated scanner (1.85 MB) differs in about 9,600 lines; the regeneration needs a 12 GB heap and took 4 min 23 s here. Without -x regenerateTlds it first downloads the current IANA TLD list and may rewrite ASCIITLD.jflex, an unrelated change.
• Edge cases now match the http rule: see ftp://ftp.example.com. and (ftp://ftp.example.com), next keep the URL whole without the punctuation; ftp:// and ftp://-x are unchanged; file://host stays split. No tokenization benchmark was run.
• The sibling bug LFV-739 changes other lines of the same grammar; one regeneration covers both, but whether the two changes compose was not checked.

What the checks validated

• Scratch worktree of a9779ef3c193, gradle, JDK 25: stock + the new test fails 5 of 6 with assertions on and off; with the fix it passes 6 of 6.
• --tests 'org.apache.lucene.analysis.email.*' (3 upstream suites + the new one): 96 tests, 1 skipped (@Nightly), 0 failures, assertions on and off. The whole :lucene:analysis:common:test: 1,915 tests, 18 skipped, 0 failures, assertions on. checkGoogleJavaFormat and regenerateUAX29URLEmailTokenizerChecksumCheck pass.
• The regenerated scanner equals the finder's independent JFlex 1.9.1 output except for its header comment. With it shadowed: the finder's test 4 of 4, the independent reproduction's 11 of 11, the cross-model reviewer's 4 of 4, and the hunt's translation-validation harness loses the {ftp, no path} failure set without gaining any.

Say so in the class and package javadoc. Not run.

Trade-offs

Changes no tokens, but contradicts RFC 1738 3.2.2, keeps the http/ftp asymmetry, and leaves the <NUM> and <EMAIL> fallbacks.

A char filter (for example a PatternReplaceCharFilter) that appends / to path-less ftp URLs. Not run.

Trade-offs

No Lucene change, but it alters the indexed text of those URLs and every application must do it itself.

Method: bounded-exhaustive translation validation (the hunt's semi-formal tier), a hunt unit (classic, Wikipedia and URL/e-mail tokenizers against their documented rules), 2026-10-27.

• An AI finder wrote an independent reference from the RFC 1738 / RFC 3986 grammar that the tokenizer cites and generated URLs from 5 schemes x 5 userinfo forms x 10 hosts x 4 ports x 19 paths x 7 queries x 5 fragments, each in 6 surrounding contexts. The completeness oracle: a valid URL between neutral delimiters comes out as exactly one <URL> token.
• Every path-less ftp case in this finite matrix failed, so {ftp, no path} came out as a minimal always-failing feature set. The same run flagged other feature sets that are separate findings, among them the sibling path-character bug (LFV-739).
• Reading the FTPurl rule gave the root cause; the finder then reduced it to a public-API JUnit. Against the fixed scanner the harness loses {ftp, no path} and gains no failing set.