StandardTokenizer misapplies the UAX #29 Hebrew quote rules next to digits and mid-word punctuation

UAX #29 has three rules for Hebrew written with ASCII quotes: WB7a keeps an apostrophe after a Hebrew letter in the word (BET', a geresh typed on a keyboard), and WB7b/WB7c keep a quotation mark between two Hebrew letters (TAV"ALEF, an acronym whose gershayim is typed as "). StandardTokenizer, the tokenizer of the default StandardAnalyzer, says it implements the UAX #29 word-break rules, and its grammar lists WB7a-WB7c for its word rule. It applies them correctly on their own, but its grammar writes "Hebrew letter + quote" as a separate piece that is glued to whatever piece follows and must start with its own Hebrew letter. So four no-space shapes come out wrong, silently, in StandardTokenizer and UAX29URLEmailTokenizer: BET'2 is one term (a TermQuery for 2 finds 0 documents), HE.SHIN' loses its apostrophe, ALEF.TSADI"LAMED and ALEF"BET"GIMEL are split, and TAV"ALEF.BET is split. ICU4J and Lucene's own ICUTokenizer give the UAX #29 answer, and the same words written with U+05F3/U+05F4 are tokenized correctly. Every release from 4.7.0 to 10.5.2, 10.6.0 (by reading) and main is affected. Folding the quote rules into the letter chain of both grammars fixes all four shapes; a reindex of affected fields is needed after the fix.

• Geresh and gershayim. Hebrew marks an abbreviation or a number-letter with a geresh (BET' = 2nd, as in the class name TET'3) and an acronym with a gershayim before its last letter (TAV SHIN"PE, תש"פ). Their Unicode characters are U+05F3 and U+05F4, but text often types them as the ASCII apostrophe U+0027 and quotation mark U+0022 (for the gershayim, the reporter of a 2010 Lucene issue estimated that 99.9% of Hebrew text files use "; not measured here).
• UAX #29 word breaks. The Unicode annex that StandardTokenizer implements. Each character has a Word_Break class (ALetter, Hebrew_Letter, Numeric, MidLetter such as :, MidNumLet such as ., Single_Quote ', Double_Quote ", ExtendNumLet _), and rules say where a word may not break: WB5 letter x letter, WB6/WB7 letter x mid-word character x letter, WB9/WB10 letter x digit, WB13a/b around _, and for Hebrew WB7a (Hebrew_Letter x ') and WB7b/WB7c (Hebrew_Letter x " x Hebrew_Letter). Everywhere else the word ends.
• The grammar. The scanner is generated by JFlex from StandardTokenizerImpl.jflex. Its word rule (:238-252) is a list of pieces glued by +: a letter chain AHLetter ((ExtendNumLet* | MidLetter) AHLetter)*, a digit chain, and the Hebrew piece HebrewLetter (SingleQuote | DoubleQuote HebrewLetter) (:239, :245). The scanner takes the longest match; characters outside every token are skipped.

1. Gluing two pieces with nothing between them is right when the left piece ends in a letter or digit (WB5, WB8-WB10). Of the pieces + glues directly, the Hebrew piece is the only one that can end in an apostrophe, so + also glues a digit chain to it (BET'2), and the outer ExtendNumLet+ loop glues _a to it (ALEF'_a). No UAX #29 rule joins an apostrophe to a following digit or underscore.
2. The Hebrew piece must start with its own Hebrew letter. In HE.SHIN' and ALEF.TSADI"LAMED, SHIN and TSADI come right after a mid-word ., so they are inside a letter chain, and a chain cannot end in . to hand the letter on. The longest match is HE.SHIN or ALEF.TSADI; the quote is skipped and LAMED starts a new token. The same happens after a mid-word :, and in ALEF"BET"GIMEL, whose BET already ends the piece ALEF"BET.
3. The piece TAV"ALEF ends at ALEF. Going on with .BET would need a letter chain that starts at ALEF, which the piece already used: [TAV"ALEF, BET].
4. Upstream's conformance file has the Hebrew rules only in their shortest form, ALEF' and ALEF"ALEF (WordBreakTestUnicode_12_1_0.java:7165, :7169), so its tests pass.
5. U+05F3 is ALetter and U+05F4 is MidLetter, so text with the Hebrew marks goes through the letter chain and is correct. UAX29URLEmailTokenizerImpl.jflex:372-386 has the same word rule.

Search with the default StandardAnalyzer over one document "class BET'2": body:2 0 hits on main, 1 with the fix; body:BET' 0 and 1; body:BET'2 1 and 0.

• Who is hit. Users of new StandardAnalyzer() (core), a bare StandardTokenizer, 42 other Lucene analyzers that construct one (for example EnglishAnalyzer, ArabicAnalyzer), and UAX29URLEmailTokenizer (analysis-common). No setting or opt-in API is involved: these classes in their default configuration are affected.
• What triggers it. Hebrew text that types the geresh as ' and the gershayim as ". For the gershayim, the reporter of LUCENE-2465 (#3539, 2010) wrote that "no user has such a character on his keyboard" and "In 99.9% of Hebrew text files, old and new, double-quotes is being used as GERSHAYIM" (their estimate); that the geresh is likewise usually typed as ' is likely but not sourced. Then only four no-space adjacencies go wrong. The most natural is a letter with a geresh followed by a digit, as in class names (TET'3, BET'2); the others need mid-word punctuation next to a quoted Hebrew word (TAV"ALEF.BET). Plain acronyms (TAV SHIN"PE) and ordinary words ending in a geresh (GIMEL'VAV RESH GIMEL') are correct; the geresh is lost when the letter before it comes right after a mid-word character such as ., :, " or U+05F4 (HE.SHIN', HE:SHIN', ALEF"BET'). How often these shapes occur in real text was not measured.
• What goes wrong. Wrong terms, silently: a merged term hides the digit from search, a dropped apostrophe conflates two forms, a split word indexes two terms. Text analyzed the same way at index and query time still matches itself, so whole-form queries work; queries for a part, exact-term lookups built another way and term statistics see the wrong terms, and so, depending on their configuration, can facets on analyzed text and highlighting (not run).
• Not more than that. A 1,200,000-character run of BET'2 tokenizes in 49-53 ms into 4,706 tokens of at most 255 characters; neighbouring tokens and the next value are unchanged.
• Long-lived, and a fix changes terms. Every release since 4.7.0 (2014). Affected fields need a reindex after a fix.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: term 0 expected:<ב'[]> but was:<ב'[2]>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:262)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:513)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:573)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:637)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertAnalyzesTo(BaseTokenStreamTestCase.java:792)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertAnalyzesTo(BaseTokenStreamTestCase.java:777)
       org.apache.lucene.tests.analysis.BaseTokenStreamTestCase.assertAnalyzesTo(BaseTokenStreamTestCase.java:862)
       org.apache.lucene.analysis.standard.TestStandardTokenizerHebrewApostrophe.testHebrewLetterApostropheThenDigit(TestStandardTokenizerHebrewApostrophe.java:26)

• The contract. "This class implements the Word Break rules from the Unicode Text Segmentation algorithm, as specified in Unicode Standard Annex #29" (StandardTokenizer.java:31-32; nearly the same, "implements Word Break rules from ...", in UAX29URLEmailTokenizer.java:31-33). The grammar comment of this rule lists WB7a, WB7b and WB7c (StandardTokenizerImpl.jflex:226-237).
• The author's statement. In LUCENE-6103 (#7165, 2014), asked about ASCII " in Hebrew, sarowe wrote: "I did follow the specs to the letter, and it does the right thing: Rules WB7b and WB7c forbid breaks around the ASCII double quote character, but only when surrounded by Hebrew letters"; later in the same thread, answering a request to tailor the colon rule, they wrote "StandardTokenizer implements UAX#29 word break rules. Done." Only the plain Hebrew"Hebrew case was discussed; it works.
• Documented deviations. The grammar explains its TR#51 emoji rules in place of WB3c/WB15/WB16 (StandardTokenizerImpl.jflex:166-208) and its Complex_Context run rule (:255-274); the conformance-test generator skips two a ZWJ ExtPict cases "because it conflicts with TR#51 v11.0 rules" (generateJavaUnicodeWordBreakTest.pl:130-140). Nothing mentions the Hebrew rules.
• Consistency. The Latin analogue abc'3 gives [abc, 3]; the U+05F3/U+05F4 spellings of the same words follow UAX #29; ICU4J 78.3 and ICUTokenizer give [class, TET', 3], [TAV"ALEF.BET] and [ALEF"BET"GIMEL] (ICUTokenizer has its own tailorings and is not an oracle for every row; it differs from ICU4J on ALEF'_a).
• Precedent. LUCENE-5447 (#6510, fixed in 4.7.0) was a composition gap in the same word rule that the conformance lines missed, fixed as a bug.
• No decision on record. The piece came with the Unicode 6.3 upgrade (LUCENE-5357, #6421), which does not discuss how it combines with the other pieces; no later commit changed it.
• Framing. UAX #29 allows tailoring and a tokenizer emits tokens, not raw segments, so this is best framed as an undocumented rule-composition error in the rules the grammar itself names, not as a claim that every token must equal an ICU4J segment.

The intended-behaviour check, the devil's-advocate verifier, built the strongest case for "intended": the javadoc invites users to copy the grammar if it does not suit them (StandardTokenizer.java:34-36); the official conformance tests pass; UAX #29 allows tailoring; in #3539 (2010) rmuir said mid-word ASCII quotes in Hebrew are the application's job ("that U+0022 should really be U+05F4. then it will work correctly with Lucene (including StandardTokenizer)"; mapping ' to U+05F3 is the same advice extended), and in the same thread described ICUTokenizer's tailored Hebrew grammar of the time (' as Extend in Hebrew context, so BET'2 stays one word) as the recommended approach; with U+05F3, UAX #29 itself keeps BET GERESH 2 as one word, which is what the tokenizer emits today for BET'2; and identical text still matches itself.

Why it fails: the invitation to copy is general, not a documented Hebrew deviation, and the grammar names WB7a-WB7c for this rule; the conformance file has only the shortest Hebrew cases; #3539 predates WB7a-WB7c (Unicode 6.3), which LUCENE-5357 adopted for the ASCII forms on purpose, and the ICUTokenizer tailoring it describes is gone (today's Default.rbbi:142-146 states WB7a-WB7c as in the annex, and ICUTokenizer splits BET'2), and with U+05F4 TAV GERSHAYIM ALEF.BET is one word while the ASCII form is split; for the ASCII BET'2 the annex, ICUTokenizer and the Latin abc'3 all break; symmetry helps only whole-form queries. Verdict: not intended (the cross-model review, the cross-model "intended" lens, rated the argument weak; the intended-behaviour check, the adjudication and audit 4 agree).

Option A was validated in a scratch worktree at a9779ef3c193 (OpenJDK 25.0.3): grammar edit, both scanners regenerated with the build's own tasks, then the repro and its twin, the earlier repros, the analysis tests of lucene/core, the whole analysis-common module, analysis.tests and the static checks, with and without assertions. Options B-D were not run. A rough timing shows at most a small difference (StandardTokenizer about 1-3% slower); no controlled (JMH) benchmark was run. Until a fix ships, a char filter that maps the ASCII quotes next to Hebrew letters to U+05F4/U+05F3 avoids shapes 2-4 (option C).

The same change in StandardTokenizerImpl.jflex and UAX29URLEmailTokenizerImpl.jflex (17 lines added, 14 removed in each), then ./gradlew :lucene:core:regenerateStandardTokenizer :lucene:analysis:common:regenerateUAX29URLEmailTokenizer (with -x :lucene:analysis:common:regenerateTlds so the TLD list is not re-downloaded). X"Y becomes one letter of the chain, so it can follow or precede a mid-word character (shapes 3, 4); an apostrophe after a Hebrew letter ends the word unless a letter follows it (WB7a; shapes 1, 2); with a letter after it, the apostrophe stays a WB6/WB7 mid-word character through MidLetterEx (GIMEL'VAV RESH GIMEL' stays one word). Every row of the table gets the UAX #29 answer in both tokenizers; the controls are unchanged.

ALetterSansHebrewEx  = \p{WB:ALetter} {ExtFmtZwj}
HebrewAtom           = {HebrewLetterEx} ( {DoubleQuoteEx} {HebrewLetterEx} )*
LetterAtom           = {HebrewAtom} | {ALetterSansHebrewEx}
LetterChain          = {LetterAtom} ( ( {ExtendNumLetEx}* | {MidLetterEx} ) {LetterAtom} )*
LetterChainHebrewEnd = ( {LetterAtom} ( {ExtendNumLetEx}* | {MidLetterEx} ) )* {HebrewAtom}
NumericChain         = {NumericEx} ( ( {ExtendNumLetEx}* | {MidNumericEx} ) {NumericEx} )*
KatakanaChain        = {KatakanaEx} ( {ExtendNumLetEx}* {KatakanaEx} )*
WordGroup            = {KatakanaChain} | ( {NumericChain} | {LetterChain} )+
WordGroupHebrewEnd   = ( {NumericChain} | {LetterChain} )* {LetterChainHebrewEnd}
// the word rule, replacing StandardTokenizerImpl.jflex:238-251:
  {ExtendNumLetEx}* {WordGroup} ( {ExtendNumLetEx}+ {WordGroup} )* {ExtendNumLetEx}*
| {ExtendNumLetEx}* ( {WordGroup} {ExtendNumLetEx}+ )* {WordGroupHebrewEnd} {SingleQuoteEx}
  { return WORD_TYPE; }

Trade-offs

• Indexed terms change for text with the four shapes (and, within the exhaustive alphabets and the module tests, nowhere else). Affected fields need a reindex; it probably belongs under "Changes in Runtime Behavior" and may be main-only.
• Generated code. StandardTokenizerImpl.java changes 43 lines of packed tables; UAX29URLEmailTokenizerImpl.java rewrites 24,701 lines of tables with the same line count; the compiled classes stay within 15 bytes of the stock ones. Regenerating the email scanner takes minutes and a 12 GB heap.
• Speed. Rough timing on about 24,000,000 characters of mixed Latin and Hebrew words, best of 5: StandardTokenizer 298-304 ms stock vs 305-308 ms fixed (about 1-3% slower), UAX29URLEmailTokenizer 330-331 vs 331-334 ms (three runs per configuration, two of them alternating, on a loaded host; the fix emits 6% more tokens on this text). No controlled (JMH) benchmark, so this shows neither a regression nor its absence.
• Two grammars carry the same rule; open #6527 (LUCENE-5464) proposes sharing rule text.

What the checks validated

• Regeneration ran cleanly and updated both checksum files.
• The repro and its twin: 5 of 6 and 4 of 5 fail on stock, 6 of 6 and 5 of 5 pass with the fix (gradle, -ea and -da). With the regenerated scanners on the stock jars, the finder's JUnit passes 7 of 7 and the independent reproduction's tests 7 of 7 and 6 of 6.
• :lucene:core:test --tests 'org.apache.lucene.analysis.*' (22 suites, 174 tests, including TestStandardAnalyzer 45 of 45 with the WordBreakTest-12.1 cases) and all of :lucene:analysis:common:test (340 suites, 1,914 tests, including TestUAX29URLEmailTokenizer, 46 tests with 1 nightly-only test skipped): 0 failed, 18 skipped, with -Ptests.asserts=true and false.
• :lucene:analysis.tests:test (TestRandomChains, TestBugInSomething, TestAllAnalyzersHaveFactories): 7 tests, 0 failed, 1 skipped.
• :lucene:core:check :lucene:analysis:common:check -x test: every regenerate checksum check, format, ecjLint, forbiddenApis and license checks pass.
• The finder's exhaustive comparison with the independent UAX #29 reference (8,108,730 strings up to length 6 over a 14-code-point word alphabet): 0 StandardTokenizer mismatches with the fix (33,321 on stock); UAX29URLEmailTokenizer keeps only 2 minimal cases of its URL rules, which are not Hebrew.

Make HebrewLetter ("HebrewLetter)* an atom of the letter chain and drop the " branch of the Hebrew piece; leave HebrewLetter ' alone. Not run.

Trade-offs

Smaller, but shapes 1 and 2 stay, and shape 1 (BET'2, TET'3) is the most natural trigger.

A char filter that turns " between Hebrew letters into U+05F4 (the 2010 advice in #3539) and, by extension, ' after a Hebrew letter into U+05F3. Not run as a filter; the shapes with the marks mapped by hand were run.

Trade-offs

Shapes 2-4 then come out as one word (executed with the marks mapped by hand), but shape 1 stays joined, because U+05F3 is ALetter and UAX #29 itself keeps BET GERESH 2 as one word. It changes the indexed characters, so queries need the same filter.

State in the javadoc that WB7a-WB7c are applied only in isolation.

Trade-offs

Cheap, but keeps a deviation from the annex the class promises, while ICUTokenizer and the U+05F3/U+05F4 forms already follow it.

Method: bounded-exhaustive translation validation (the hunt's semi-formal tier; no model checker, SMT solver or proof).

• The reference. A hunt unit on UAX #29 word boundaries wrote a word-break reference for Unicode 12.1 from the annex rules, driven only by the Unicode 12.1 data files. It had 0 mismatches on all 1,823 lines of the official WordBreakTest-12.1.0, killed 6 single-rule mutants, and had 0 mismatches against ICU4J 78.3 on 168,420 strings.
• The comparison. Every string of length 1-4 over three alphabets (27, 24 and 14 code points, the last supplementary-plane) and of length 1-6 over a 14-code-point word-rule alphabet (8,108,730 strings) was tokenized by the stock StandardTokenizer and UAX29URLEmailTokenizer and compared with the reference. Mismatches were reduced to minimal strings and classified: every Hebrew mismatch is one of four shapes (98, 186, 54 and 36 minimal strings for StandardTokenizer), first seen at length 3 (2026-10-27).
• Root cause and repro. A read of the word rule found the stand-alone Hebrew piece; the finder wrote a public-API JUnit and a fix, and re-ran the exhaustive comparison with the fix (0 StandardTokenizer mismatches).
• Checks. Claude verifiers with their own tests (the independent reproduction with an ICU4J oracle, the intended-behaviour check a devil's advocate with ICUTokenizer and ICU4J side by side), review records by a model from another family (OpenAI GPT-5.6) that ran their own tests, an adjudication with fresh seeds, a registry re-run and an independent audit with its own ICU4J probe.