CSVUtil.parse decodes every quoted field except the last one, so a quoted kuromoji part-of-speech keeps its quotes

CSVUtil (public, in the exported package org.apache.lucene.analysis.util of analysis-common) is Lucene's "Utility class for parsing CSV text". Its parse splits a line at commas outside quotes and decodes each field that ends at such a comma (enclosing quotes removed, "" turned into "), but adds the last field without decoding. The same quoted value therefore decodes in the first or a middle position and keeps its quotes in the last: parse("\"p,q\",\"p,q\",\"p,q\"") returns [p,q, p,q, "p,q"], and parse does not decode its own encoder's output (quoteEscape) when it is the last field.

Kuromoji's UserDictionary.open, used by JapaneseTokenizerFactory's userDictionary option, reads every line with parse and takes the part-of-speech from the fourth and last field. An entry written with every field quoted gives tokens whose part-of-speech is "bar" with the quotes, while surface, segmentation and readings decode; a JapanesePartOfSpeechStopFilter with stop tag bar then removes nothing. A part-of-speech that contains a comma has to be quoted, so it keeps its quotes too; the only way around it is an undocumented trick, a trailing comma (an empty fifth field, which UserDictionary ignores). The line dates from the 2012 Kuromoji donation; every release from 3.6.0 to 10.6.0 and main is affected. The fix is one line.

• CSVUtil: "Utility class for parsing CSV text" (CSVUtil.java:23), a public final class in the exported package org.apache.lucene.analysis.util (analysis-common module-info.java:88), not marked internal. parse is "Parse CSV line", "@param line line containing csv-encoded data", "@return Array of values" (:35-40); quoteEscape is "Quote and escape input value for CSV" (:94) and wraps any value that holds a comma in quotes (:101-103).
• Kuromoji user dictionary: a text file of lines surface,segmentation,readings,part-of-speech (layout comment UserDictionary.java:58; the test file userdict.txt), loaded by UserDictionary.open(Reader) or the userDictionary option of JapaneseTokenizerFactory (JapaneseTokenizerFactory.java:143-155). Its entries override the system dictionary for matching text.

1. parse walks the line once and toggles insideQuote at every " (CSVUtil.java:46-52).
2. At a comma outside quotes it decodes the field: value = unQuoteUnEscape(value); result.add(value); (:54-58). unQuoteUnEscape strips enclosing quotes when the field matches ^"([^"]+)"$ and turns "" into " (:75-92).
3. After the loop it adds the rest of the line: result.add(sb.toString()); (:65). That is the last field, never decoded.
4. UserDictionary.open calls CSVUtil.parse(line) for each line (UserDictionary.java:67) and stores String pos = values[3]; (:100), the last of the 4 fields, next to each segment's reading (:130). getPartOfSpeech returns it unchanged.
5. JapanesePartOfSpeechStopFilter keeps a token unless stopTags.contains(pos) (:44), an exact match, so "bar" does not match bar.

• Who is exposed. (a) Kuromoji users with a custom user dictionary (opt-in: JapaneseTokenizerFactory's userDictionary option, UserDictionary.open, or a JapaneseAnalyzer built with one) whose part-of-speech field is quoted. (b) Any direct caller of the public CSVUtil.parse whose data quotes the last field.
• What the damage looks like. The part-of-speech tag carries literal quotes, with no error. Segmentation, readings and matching are right, so tokenization looks fine. Anything that compares tags exactly sees a different tag: JapanesePartOfSpeechStopFilter (also inside JapaneseAnalyzer, JapaneseAnalyzer.java:112) keeps tokens it should drop, and code reading PartOfSpeechAttribute gets "bar".
• Only an undocumented workaround. A part-of-speech that contains a comma has to be quoted (unquoted, the comma splits the field), and as the last field it keeps its quotes ("tagA,tagB" comes back with its quotes). The one way around it is a trailing comma: the quoted part-of-speech is then followed by a comma, so parse decodes it, and UserDictionary ignores the empty fifth field (it reads only values[0..3] and checks no field count, UserDictionary.java:95-100). On stock main, abcd,a b cd,foo1 foo2 foo3,"tagA,tagB", gives the part-of-speech tagA,tagB, and the stop tag tagA,tagB removes all 3 tokens. Nothing documents this, and it works only because of the same position rule.
• How likely. Rare: the upstream test dictionary leaves the last field unquoted, the system dictionary shipped in the kuromoji jar has no quote character in any field, and nobody has reported it since 2012.
• Scale. One line reproduces it; assertions on and off behave the same; nothing depends on index size or concurrency.
• Not affected. The kuromoji system dictionary (an enumeration of all 392,127 words on main finds no quote character in any field); nori's runtime user dictionary, which splits on whitespace. The kuromoji and nori TokenInfoDictionaryBuilder also call parse and would be affected only by a custom dictionary source that quotes its last field; kuromoji's writePosDict re-parses quoteEscape output whose last part is the inflection form, so it is wrong when that field holds a comma or a quote; UnknownDictionaryBuilder appends ,*,* before parsing, so the source's last field is decoded. None of these builder paths was run.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => arrays first differed at element [1]; expected:<[bar]> but was:<["bar"]>
       org.junit.internal.ComparisonCriteria.arrayEquals(ComparisonCriteria.java:78)
       org.junit.internal.ComparisonCriteria.arrayEquals(ComparisonCriteria.java:28)
       org.junit.Assert.internalArrayEquals(Assert.java:534)
       org.junit.Assert.assertArrayEquals(Assert.java:285)
       [...]
     Caused by: org.junit.ComparisonFailure: expected:<[bar]> but was:<["bar"]>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.junit.internal.ExactComparisonCriteria.assertElementsEqual(ExactComparisonCriteria.java:8)
       org.junit.internal.ComparisonCriteria.arrayEquals(ComparisonCriteria.java:76)
       [...]

• Contract. "Utility class for parsing CSV text" (CSVUtil.java:23); parse: "@param line line containing csv-encoded data", "@return Array of values" (:35-40); quoteEscape: "Quote and escape input value for CSV" (:94), which wraps values with a comma in quotes. No javadoc literally promises RFC 4180 or parse(quoteEscape(v)) == v; the case rests on the class's own decoding of every other field.
• Tests. TestCSVUtil tests only quoteEscape (3 tests, added by SOLR-9413). Every in-tree parse call in tests uses unquoted lines (kuromoji and nori TestUnknownDictionary), and the kuromoji userdict.txt has no quote characters. Nothing pins the quotes on the last field.
• History. The line came with the 2012 Kuromoji donation and never changed: six later commits edited parse, none touched it, and no branch ever had the fix. The comparable one-line CSVUtil omission (quoteEscape discarding the result of String.replace) was reported (#6913 / LUCENE-5851) and fixed as a bug (SOLR-9413).
• Committers on the format. In LUCENE-7635 (#8686, 2017) hossman wrote that "it seems like the intent of the UserDict format is to be \"CSV with '#' comments\"", with the example line "quoted#sharp",other,"quoted,stuff", whose last field is quoted. It was a tentative reading in a discussion about '#' comments and says nothing about how the last field is decoded. In LUCENE-8596 (#9642) msokolov called a user-dictionary parsing limitation "a real problem we should fix" and fixed it on main with a CHANGES note.
• What is not claimed. Quote-only values (quoteEscape writes them without enclosing quotes, which is not valid RFC 4180) and empty quoted fields: in CSVUtil's dialect "" is quoteEscape's encoding of one quote, which RFC 4180 reads as an empty field. A quoted field that also holds a quote keeps its enclosing quotes in every position; that is the sibling defect LFV-889.

The strongest "intended" case (rated moderate by the intended-lens review from another model family; also argued by the devil's-advocate verifier): CSVUtil began in 2012 as a narrow helper for MeCab dictionary lines, whose last field is never quoted in stock data; its javadocs are one line each and claim no RFC 4180; the raw append survived 14 years, a nori copy and two package moves without a report; the separate atilika/kuromoji project, by the original Kuromoji authors, has the same omission and its tests quote only non-last fields; and a change may break callers that rely on the raw last field.

Why it fails: the class itself decodes quoted fields in every other position, so a narrow dialect cannot explain a rule that depends only on position. Age and the inherited copy show the case was never exercised, not that it was meant: nothing documents, tests or comments on keeping the quotes, and no Lucene code relies on the raw last field. Compatibility decides how a fix ships (main, with a CHANGES entry, as LUCENE-8596 did), not whether this is a bug.

Option A is the bug's own fix. Option B adds LFV-889's recommended parse-only fix for the separate unquote defect (a quoted field that also holds a quote, in any position); it was checked with the standalone runner only. Do not use the regex ^"(.*)"$ for LFV-889: it breaks the one-quote encoding "" that quoteEscape writes. Add the new tests (or their cases in TestCSVUtil and kuromoji's TestUserDictionary) with whichever is chosen.

Line 65 of CSVUtil.java gets the call that line 56 already makes for every comma-terminated field (+1/-1). The finder's fix.

-    result.add(sb.toString());
+    result.add(unQuoteUnEscape(sb.toString()));

Trade-offs

• The smallest change, and parse no longer depends on position.
• It changes the parsed value of lines whose last field is quoted (the quotes go) or holds a doubled quote: an unquoted x,a""b now gives a"b, as a""b,x already gives in the first position. Kuromoji part-of-speech tags of such entries change; indexed terms change only where the tag decides something (a part-of-speech stop filter, or code that turns tags into terms), and those fields need a reindex. Main with a CHANGES entry, as LUCENE-8596 did; a backport is debatable.
• With A, x,"" gives [x, "], the one quote that "",x already gives in the first position: in CSVUtil's dialect "" is how quoteEscape writes one quote. A leaves the sibling LFV-889: a quoted field that also holds a quote keeps its enclosing quotes in every position (x,"say ""hi""" gives [x, "say "hi""]), and LFV-889's tests still fail 4 of 9. In the finder's bounded model with A, the round-trip law fails only for values with both a comma and a quote (10,110 of 35,140 field lists, down from 26,700).

What the checks validated

• Scratch worktree at a9779ef3c193, gradle, JDK 25: both new tests fail on stock (3 of 3 and 2 of 3, assertions on and off) and pass with the fix; org.apache.lucene.analysis.util.* (9 suites, 35 tests, TestCSVUtil included) plus the whole kuromoji (31 suites, 198 tests) and nori (13 suites, 87 tests) suites: 320 tests, 0 failures, 11 skipped, with assertions on (seed 5EED0895) and off (seed 5EED0896); the whole :lucene:analysis:common:test: 340 suites, 1,912 tests, 0 failures, 18 skipped; checkGoogleJavaFormat passes for both modules.
• Standalone runner with the fix shadowed: 3 of 3 and 3 of 3 pass with -ea and -da; earlier the finder's, the independent reproducer's, the cross-model reviewer's and the devil's advocate's tests passed with the same line. An independent fact check repeated the 320-test set with new seeds in both modes and the format check, with the same result.

Option A plus LFV-889's recommended parse-only change to unQuoteUnEscape (its independent reproducer's sketch): strip the enclosing quotes only when the field ends with a quote and begins with an odd run of quotes, then unescape. quoteEscape doubles every quote, so an unenclosed value has only even runs and "" stays one quote. Then x,"say ""hi""" gives [x, say "hi"], while x,"" and "",x keep the one-quote reading ([x, "], [", x]).

     // Unquote
     if (result.indexOf('\"') >= 0) {
-      Matcher m = QUOTE_REPLACE_PATTERN.matcher(original);
-      if (m.matches()) {
-        result = m.group(1);
+      // quoteEscape doubles every quote, so an unenclosed value has only even-length quote runs;
+      // an enclosed value starts with an odd-length run (the enclosing quote plus doubled quotes).
+      int lead = 0;
+      while (lead < result.length() && result.charAt(lead) == QUOTE) {
+        lead++;
+      }
+      if (result.length() >= 2
+          && (lead & 1) == 1
+          && result.charAt(result.length() - 1) == QUOTE) {
+        result = result.substring(1, result.length() - 1);
       }
...
-    result.add(sb.toString());
+    result.add(unQuoteUnEscape(sb.toString()));

Trade-offs

It changes the output for quoted fields that hold a quote, in every position, so it needs its own CHANGES note and is LFV-889's decision too. The sketch leaves QUOTE_REPLACE_PATTERN and the Matcher import unused. Do not use the finder's first sketch for LFV-889, the regex ^"(.*)"$, instead: it reads "" as an empty field, so "",x gives [, x] instead of [", x], and the round-trip law then fails for 3,578 field lists, 2,316 of them in non-last fields that pass on stock; LFV-889's judge says it must not be proposed.

What the checks validated

Standalone runner on a9779ef3c193 with the combined source shadowed (not gradle, not the module suites, not format-checked): the brief's tests pass 3 of 3 and 3 of 3 with -ea and -da; LFV-889's own tests pass 9 of 9 (4 of 9 fail on stock and with A alone); the finder's bounded model finds 0 of 35,140 round-trip mismatches and 1,713 of 18,304 RFC 4180 reference mismatches, all of the quoted-empty or escaped-quote shape, where RFC 4180 reads "" differently from CSVUtil's dialect.

Say so in the parse javadoc. Not run.

Trade-offs

Changes no output, but documents a position-dependent rule that no CSV writer produces, leaves a part-of-speech with a comma decodable only through the trailing-comma trick, and contradicts quoteEscape's own output.

Method: bounded-exhaustive translation validation on the stock main jars (the study's "semi-formal" tier). A hunt unit, a sweep of public analysis-common APIs that no earlier hunt had named, 2026-10-28.

• Reference check: all 87,381 lines of up to 8 symbols over {a, comma, quote, U+10400} (18,304 valid under RFC 4180) were parsed by the stock CSVUtil and by a reference parser. All 5,811 mismatches are quoted fields. In the last field the stock code returns the field raw: plain 2,814 (this bug alone), empty 449 and escaped quote 962. The empty and escaped-quote shapes also mismatch in non-last fields (1,122 and 464), where RFC 4180 and CSVUtil's dialect read "" differently or the sibling LFV-889 applies, so only the plain count is this bug's.
• Round-trip check: parse(join(quoteEscape(f_i))) == fields over 35,140 field lists. The classes "last field with a comma" (8,295) and "last field with a comma and a quote" (4,206, which also needs LFV-889's fix) come from this bug; the last-field quote-only class (8,295) comes from quoteEscape's unenclosed encoding of quote-only values and is not claimed (the cross-model reviewer's correction).
• Re-run for this doc with the fixes shadowed: option A alone leaves only the round-trip failures of values with both a comma and a quote (10,110, LFV-889) and reference mismatches of the same two shapes in every position (2,997); A plus LFV-889's parse-only fix leaves 0 round-trip failures and 1,713 reference mismatches, all where RFC 4180 reads "" differently from CSVUtil's dialect.
• Then an AI finder wrote the public-API JUnits, and independent AI tools reproduced, reviewed, judged and audited it (see the checks).