ConcatenateGraphFilter treats a custom tokenSeparator as its internal one-byte separator label: an ASCII separator is doubled wherever it already occurs inside a token ('e-mail' becomes 'e--mail'), and a non-ASCII separator fails an assertion with -ea or is written as one truncated byte with -da (U+2013 becomes U+0013, U+00B7 a lone 0xB7 byte, i.e. invalid UTF-8)
Finding LFV-517Area: analysis-common / ConcatenateGraphFilter and ConcatenateGraphFilterFactory (custom tokenSeparator)Priority: lowImpact: silent wrong resultsNovelty: new
AI-assisted finding, confirmed by automated checks, not yet triaged by a human Lucene committer. This site files nothing upstream.
What it shows
You join tokens with ConcatenateGraphFilter (or the concatenateGraph factory) and a custom tokenSeparator, and expect what the javadoc says, every token joined with that separator: the single token 'e-mail' stays 'e-mail', 'e-mail' and 'x' joined with '-' give 'e-mail-x', and 'new' and 'york' joined with U+2013 give the UTF-8 bytes of new, U+2013, york. Instead the filter passes the custom separator to an escaper written for its internal one-byte label U+001F, so every occurrence of an ASCII separator inside a token is doubled ('e--mail', 'e--mail-x', and 'new york' for a keyword token with a space separator). A non-ASCII separator fails the escaper's assertion with -ea and, with -da, is cut to its low byte: U+2013 gives 'new' U+0013 'york', and U+00B7 gives the invalid UTF-8 term [6e 65 77 b7 79 6f 72 6b].
The test shows that ConcatenateGraphFilter with the custom separator U+2013 (EN DASH) treats it as its internal one-byte separator label when joining new and york.
Expected vs actual
Expected: the term bytes are the UTF-8 of new–york, [6e 65 77 e2 80 93 79 6f 72 6b]. Actual: with -ea an AssertionError inside the filter (ConcatenateGraphFilter.java:260); with -da [6e 65 77 13 79 6f 72 6b] (U+2013 cut to 0x13).
Files
src/org/apache/lucene/analysis/miscellaneous/TestConcatenateGraphNonAsciiSeparator.java(JUnit test)
Run it
- Tested on Lucene main e1faeba08bde (2026-11-02): still reproduces.
- Tested on Lucene main 645d4fe14a20 (2026-10-28): reproduced.
- Recorded outcome of run 1 on Lucene main 645d4fe14a20: 1 test(s) run, 1 failed.
- How to read the outcome: a failing test is the finding reproducing (the failure message states expected against actual).
- Seeds: a rerun with the same seed replays the same random choices. Assertions off (-da -dsa -Dtests.asserts=false) is the production-like setting.
# Lucene checkout at the tested commit, built with: ./gradlew --no-daemon assemble (JDK 25)
LUCENE=/path/to/lucene; V=12.0.0-SNAPSHOT
CP=$LUCENE/lucene/core/build/libs/lucene-core-$V.jar
CP=$CP:$LUCENE/lucene/codecs/build/libs/lucene-codecs-$V.jar
CP=$CP:$LUCENE/lucene/test-framework/build/libs/lucene-test-framework-$V.jar
CP=$CP:$LUCENE/lucene/analysis/common/build/libs/lucene-analysis-common-$V.jar
CP=$CP:randomizedtesting-runner-2.9.2.jar:junit-4.13.2.jar:hamcrest-3.0.jar # test-framework dependencies (Maven Central)
CP=$CP:junit-platform-console-standalone-6.1.3.jar # JUnit Platform console launcher (Maven Central)
javac -d classes -cp "$CP" $(find src -name '*.java')
# run 1: assertions on, seed DEADBEEF
java -ea -esa --add-modules jdk.incubator.vector --enable-native-access=ALL-UNNAMED -Dtests.seed=DEADBEEF \
-cp "classes:$CP" org.junit.platform.console.ConsoleLauncher \
execute --select-class org.apache.lucene.analysis.miscellaneous.TestConcatenateGraphNonAsciiSeparatorGradle alternative: copy the test into the matching module's src/test/ tree of the checkout and run ./gradlew -p lucene/<module> test --tests <class> -Ptests.seed=<seed> (add -Ptests.asserts=false for a run without assertions).
Feedback
Open an issue in lucene-ai-bug-hunt. Please do not open issues or pull requests in apache/lucene about this finding unless a Lucene committer agrees it should be filed.
Snapshot of 2026-11-04. AI-assisted; not yet triaged by a human Lucene committer. Feedback: open an issue in lucene-ai-bug-hunt.