Lowering StandardTokenizer.setMaxTokenLength between two incrementToken() calls makes the next call throw ArrayIndexOutOfBoundsException

StandardTokenizer.setMaxTokenLength also resizes the scanner's internal text buffer (StandardTokenizer.java:101-104 at 5adff78bb168), and StandardTokenizerImpl.setBufferSize allocates a new char[length], copies min(old, new) chars and keeps every read position (StandardTokenizerImpl.java:713-718). Lowered between two incrementToken() calls, below the text the scanner has already read, those positions point past the end of the smaller buffer, and the next incrementToken() throws ArrayIndexOutOfBoundsException (from zzRefill, :759) or IndexOutOfBoundsException (from Character.codePointAt, :995), with assertions on and off. The setter's javadoc allows any length in [1, 1048576] and says nothing about when it may be called. Raising the length mid-stream works, and so does lowering it before reset() or between streams; StandardAnalyzer, UAX29URLEmailAnalyzer and the factories set it before reset(), so text sent through them cannot trigger it. UAX29URLEmailTokenizer has the identical code and fails the same way. The resize came with LUCENE-5897 (4.10.0, and the 4.9.1 bugfix release); 114 releases from there to 10.6.0 fail, 4.0.0 to 4.9.0 pass. Resizing only when the buffered text still fits, and otherwise leaving the smaller size to the next reset(), fixes it. Priority Low.

1. The setter resizes the buffer. After the range check: if (length != maxTokenLength) { maxTokenLength = length; scanner.setBufferSize(length); } (StandardTokenizer.java:101-104). The scanner's buffer starts at 255 chars (StandardTokenizerImpl.java:51, :621).
2. The resize keeps the positions (the fix site). setBufferSize allocates new char[numChars], copies Math.min(zzBuffer.length, numChars) chars and leaves zzStartRead, zzMarkedPos, zzCurrentPos and zzEndRead unchanged (StandardTokenizerImpl.java:713-718; source StandardTokenizerImpl.jflex:151-156).
3. The repro. Reading hello left zzEndRead = 5 and zzMarkedPos = 5; the new buffer is char[2]. The next getNextToken sets zzStartRead = 5 (:980), finds no buffered char left and calls zzRefill (:1004), which first moves the unread part to the front: System.arraycopy(zzBuffer, zzStartRead, zzBuffer, 0, zzEndRead - zzStartRead) (:756-759). A source offset of 5 in a 2-char array throws.
4. With unread text left, getNextToken reads Character.codePointAt(zzBuffer, zzCurrentPos, zzEndRead) with zzEndRead past the array (:995): IndexOutOfBoundsException with a null message, and the rest of the stream is lost.
5. Why other call orders work. Before reset(): reset() calls yyreset (StandardTokenizer.java:193-198), which resets every position (StandardTokenizerImpl.java:832-840). Right after reset(): nothing is buffered yet. After incrementToken() has returned false, or after end(): that call already compacted the buffer (zzRefill, StandardTokenizerImpl.java:756-765), so every position is 0 (zzEndRead = 0) and the end-of-file flag is set; the scanner returns end-of-stream without reading the buffer (executed). Right after the last token, before that call, is still the crash window: the repro does exactly that. Between streams: close() and reset() reset the positions. A new length at least as large as the buffered text (hello world, 11 chars, lowered to 20) and any raise also work.

Window: only between an incrementToken() that returned true and the next call, and only when the new length is below the scanner's high-water mark, the chars it has buffered (zzEndRead + zzFinalHighSurrogate).

Same code elsewhere: UAX29URLEmailTokenizer (analysis-common) has the identical setter and setBufferSize (UAX29URLEmailTokenizer.java:100-110, UAX29URLEmailTokenizerImpl.java:33585-33590) and fails the same way; the suggested fix covers both. ClassicTokenizer's setter only checks the value and sets a field; it resizes no scanner buffer (ClassicTokenizer.java:83-89) and is not affected.

• Who. Code that holds one of the two tokenizers directly and lowers setMaxTokenLength between two incrementToken() calls of one stream, below the text the scanner has already read. No in-tree caller or test does this, and the verifiers found no real use.
• Default vs opt-in. Not on any default path. StandardAnalyzer sets the length right after construction and again in its setReader hook, before reset() (StandardAnalyzer.java:87, :92-93); UAX29URLEmailAnalyzer does the same (UAX29URLEmailAnalyzer.java:92, :97-98); both factories set it in create(). Calling StandardAnalyzer.setMaxTokenLength mid-stream only changes a field that the next tokenStream() applies (the devil's-advocate verifier's probe).
• Effect. One unchecked exception from the next incrementToken(); the rest of that stream is not tokenized. No wrong tokens were seen in any run. close() and a new reader reset the positions.
• Scale. Any text longer than the new length that the scanner has already read; 5 characters are enough. Same with assertions on and off.
• Likelihood. Contrived: the recorded searches found no public report in the twelve years since 4.10.0.

TestStandardTokenizerMaxTokenLengthMidStream (one method, extends LuceneTestCase, package org.apache.lucene.analysis.standard of the lucene/core tests). The test draws no random values; every seed tried failed. checkGoogleJavaFormat accepts the file.

cp $REPRO/TestStandardTokenizerMaxTokenLengthMidStream.java $LUCENE/lucene/core/src/test/org/apache/lucene/analysis/standard/
cd $LUCENE && ./gradlew :lucene:core:test --tests org.apache.lucene.analysis.standard.TestStandardTokenizerMaxTokenLengthMidStream -Ptests.seed=DEADBEEF
# and with -Ptests.asserts=false: the same failure

TestStandardTokenizerMaxTokenLengthMidStream > testLowerMaxTokenLengthMidStream FAILED
    java.lang.ArrayIndexOutOfBoundsException: arraycopy: last source index 5 out of bounds for char[2]
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:FB5A7A5A97DBDDA8]:0)
        at java.base/java.lang.System.arraycopy(Native Method)
        at org.apache.lucene.analysis.standard.StandardTokenizerImpl.zzRefill(StandardTokenizerImpl.java:759)
        at org.apache.lucene.analysis.standard.StandardTokenizerImpl.getNextToken(StandardTokenizerImpl.java:1004)
        at org.apache.lucene.analysis.standard.StandardTokenizer.incrementToken(StandardTokenizer.java:157)
        at org.apache.lucene.analysis.standard.TestStandardTokenizerMaxTokenLengthMidStream.testLowerMaxTokenLengthMidStream(TestStandardTokenizerMaxTokenLengthMidStream.java:31)
1 test completed, 1 failed

# with -Ptests.asserts=false: the same exception and frames
# 10.6.0 release jars, assertions off: the same message at zzRefill(StandardTokenizerImpl.java:751)

• Contract. The setter's javadoc names one failure, IllegalArgumentException outside [1, MAX_TOKEN_LENGTH_LIMIT] (StandardTokenizer.java:86-94). The TokenStream workflow (TokenStream.java:49-62) does not forbid reconfiguring between incrementToken() calls.
• Broken state, not a check. The setter accepts the call; the exception comes later, from internal positions that point past the scanner's own buffer.
• Siblings and history. Raising works. ClassicTokenizer's setter (a range check and a field, no buffer resize) works mid-stream. Before acf92428507 (4.10.0) the setter was a range check plus a field assignment and 4.9.0 passes the repro. The resize came with a speed fix (LUCENE-5897, #6959), whose discussion never mentions calls during a stream.
• Precedent. LUCENE-5235 (34adebab3b8, 2013) replaced tokenizers' "NullPointerException or ArrayIndexOutOfBoundsException on best effort which was not user-friendly" with an IllegalStateException.
• The repro's answer does not depend on how a new limit should apply: no text is left, so incrementToken() must return false.

The strongest case for "intended": tokenizers are configured between streams by convention. Every in-tree caller sets the length before reset(); the upstream tests that call the tokenizer's setter do so after close() or right after construction (TestStandardAnalyzer.java:100-108, TestUAX29URLEmailTokenizer.java:65-71, :110-112); ClassicAnalyzer's javadoc says "This setting only takes effect the next time tokenStream or tokenStream is called" (ClassicAnalyzer.java:78). A maintainer may fairly answer "set it before reset()".

Why it fails: the convention is not written on the tokenizer, whose public setter accepts an in-range value and leaves the object broken instead of rejecting the call. It does make "document it and reject cleanly" (fix B) as acceptable as making the call work (fix A). The devil's-advocate verifier judged the claim real at Low; the cross-model reviewer rated the intended-behaviour argument weak.

Fix A was built through gradle on 5adff78bb168, regenerated from the .jflex sources, and checked; fix B's check was run only in the lifecycle probe.

In setBufferSize of StandardTokenizerImpl.jflex and UAX29URLEmailTokenizerImpl.jflex (then regenerate the .java files and checksums), keep the copy only when numChars >= zzEndRead + zzFinalHighSurrogate. yyreset already replaces a buffer larger than ZZ_BUFFERSIZE (StandardTokenizerImpl.java:837-839), so a deferred shrink takes effect at the next reset() or close().

   public final void setBufferSize(int numChars) {
     ZZ_BUFFERSIZE = numChars;
     // Resize now only if the input already buffered for the current stream still fits;
     // otherwise yyreset() applies the new size when the next stream starts.
     if (numChars >= zzEndRead + zzFinalHighSurrogate) {
       char[] newZzBuffer = new char[ZZ_BUFFERSIZE];
       System.arraycopy(zzBuffer, 0, newZzBuffer, 0, Math.min(zzBuffer.length, ZZ_BUFFERSIZE));
       zzBuffer = newZzBuffer;
     }
   }

Trade-offs

Whenever the buffered text fits, fix A runs exactly the statements main runs today (the same copy), so those call orders are unchanged by construction; it differs only when the text does not fit, the crash window. In the crash window the rest of the current stream keeps the old, larger buffer, so a later token longer than the new limit is skipped by the tokenizer's length check (StandardTokenizer.java:163-173) instead of chopped; from the next reset() on it is chopped as documented. Executed: the lifecycle probe on stock and fix A differs only in the two rows that crash on stock (abcdefghij lowered to 3 after the first token now ends normally; aaaaaaaaaa bbbbbbbbbb lowered to 4 gives [aaaaaaaaaa]). Chopping the rest of the stream as well would need the unread text moved to the front first, as zzRefill does; that variant was not written.

What the checks validated

• Gradle in a scratch worktree at 5adff78bb168, seed DEADBEEF: without the fix the repro fails (assertions on and off), and so do the same steps on UAX29URLEmailTokenizer (a check file, not part of the repro).
• The .jflex edit was regenerated with the build's own JFlex (regenerateStandardTokenizer, regenerateUAX29URLEmailTokenizer); each regenerated .java differs from main only in this method.
• With the fix the repro passes with assertions on and off; with assertions on, every core org.apache.lucene.analysis.standard test and every analysis-common org.apache.lucene.analysis.email test pass (including TestStandardAnalyzer, TestUAX29URLEmailTokenizer and the sibling check): 140 tests, 0 failures, 1 skipped. The generated-sources checksum checks and :lucene:core:checkGoogleJavaFormat pass. The analysis-common format check (for the regenerated UAX29URLEmailTokenizerImpl.java) and the full module suites were not run, and only OpenJDK 25.0.3 on Linux aarch64 was used.

Have setBufferSize throw IllegalStateException when numChars < zzEndRead + zzFinalHighSurrogate, assign maxTokenLength only after the scanner accepted, and add a javadoc sentence to both setters.

Trade-offs

Follows the convention and the LUCENE-5235 precedent and is easy to argue. The check fires only in the crash window: once incrementToken() has returned false every scanner position is 0, so lowering the length after the stream or after end() still works. Executed by the fact check with this check shadowed in (lifecycle probe, assertions off): only the two rows that crash on stock throw IllegalStateException; afterExhausted, afterEnd and every other row are unchanged. Set maxTokenLength only after the scanner accepted, or getMaxTokenLength() reports the rejected value. The regression test would assert the exception instead.

What the checks validated

Lifecycle probe only, with the check shadowed in (fact check, assertions off); not run through gradle.

The hunt's first sketch: setBufferSize grows at once and leaves every shrink to yyreset. It passed the repro and TestStandardAnalyzer (45 tests) there.

Trade-offs

It also changes a call order that works today: lowering after reset() but before the first token chops at the new length now ([abc, def, ghi, j], probe row afterReset) and would then skip the long token instead (by reading the code; not run). Fix A keeps that case unchanged.

What the checks validated

The hunt's shadowed runs only (repro and TestStandardAnalyzer, assertions on).

Method: semi-formal (a proof-obligation audit of a class's contract, witnessed by an executable repro).

• The finder treated the setter's javadoc (only a range check) and the TokenStream workflow as obligations: an accepted call must keep the scanner's positions inside its buffer. setBufferSize keeps every position while it can cut the buffer below them.
• The finder wrote a public-API JUnit test (abcdefghij, lowered to 3), ran it on main and the 10.6.0 and 9.1.0 jars, and a probe showed the IndexOutOfBoundsException variant and the working controls (raising mid-stream, shrinking between streams).
• An independent reproducer, an executed cross-model review and a devil's-advocate verifier reproduced it with their own tests, and an adjudication re-ran them ("Prior reports, history and checks"); the reproducer's smaller test (hello, lowered to 2) became the repro.