MappingCharFilter.reset() keeps the first read's offset corrections, so offsets are wrong after a re-read

MappingCharFilter overrides Reader.reset() so that it can be rewound and read again. reset() rewinds the input and the read position, but not the list of offset corrections the filter inherits from BaseCharFilter. On the second read every length-changing match computes its correction from the first read's last running total and appends it to the old list.

So after reset() and a re-read, correctOffset() is wrong: for a->xy over "a" it maps the end of the output to 0 instead of 1, and a tokenizer over "a b" gives xy[0,0] b[1,2] instead of xy[0,1] b[2,3]. For deleting or shrinking maps, offsets come out too large, often past the end of the input (assertions off); with assertions on, BaseCharFilter's ordering assertion can fire inside read() and the second read never finishes. When the re-read completes, characters, terms and matching are right. No Lucene code path resets a char filter, so only application code that calls reset() itself and reads again is hit. The fix is to clear the list in reset() (8 added lines), or to drop reset() support.

• BaseCharFilter.java:31-33: the correction list is private int[] offsets; private int[] diffs; private int size. It is private, and BaseCharFilter has no method that clears it.
• BaseCharFilter.java:55-57: getLastCumulativeDiff() returns diffs[size - 1], the running total of the last entry.
• MappingCharFilter.java:142-157: for a match whose replacement has a different length, prevCumulativeDiff = getLastCumulativeDiff() (:145), and the new entries are placed and valued relative to it (:148 for a shorter replacement, :153-156 for a longer one).
• MappingCharFilter.java:62-68: reset() is input.reset(); buffer.reset(input); replacement = null; inputOff = 0;. The list is kept.
• BaseCharFilter.java:59-67: addOffCorrectMap documents that each offset "will not be smaller than the offset given at the previous invocation"; :77-87 asserts it. correct() binary-searches the list (:46), which needs it sorted.

pass 1   running total  0   adds (off 1, diff -1)    list [(1,-1)]          correctOffset(2) = 2-1 = 1
reset()  input rewound, inputOff = 0                 list [(1,-1)]   kept
pass 2   running total -1   adds (off 2, diff -2)    list [(1,-1),(2,-2)]   correctOffset(2) = 2-2 = 0

For b->"" over "ba": pass 1 adds (0, +1), so correctOffset(0..1) = [1,2]. Pass 2 starts from +1 and computes (-1, +2), an offset that goes backwards. With assertions on, addOffCorrectMap throws AssertionError: Offset #1(-1) is less than the last recorded offset 0 from inside read(); with them off the list [0,-1] is unsorted, the binary search for 1 lands on the entry (-1, +2), and correctOffset(1) = 1 + 2 = 3, past the end of the 2-character input.

The characters stay right whenever the re-read completes: the characters read() returns depend only on state that reset() rewinds (input, buffer, replacement, inputOff) and on the immutable map. The stale list is read by correct() and, inside read(), by getLastCumulativeDiff(), whose value only goes into new corrections; those go through addOffCorrectMap, so with assertions on its ordering assertion can throw from inside read() (the b->"" case) before the second read finishes.

• Trigger. An explicit reset() on a MappingCharFilter over a reader that can rewind (such as StringReader), a second read (with assertions on, that read can already throw), and then offsets. The use reset() support was added for in 2009 was exactly this: show the normalized text, rewind, tokenize the same reader.
• Not on any Lucene path. Analyzer calls initReader for every value (Analyzer.java:155, :191), MappingCharFilterFactory.create builds a new filter each time (MappingCharFilterFactory.java:91), and Tokenizer never rewinds its reader. MappingCharFilter is the only CharFilter in the source tree that overrides reset(); the others inherit java.io.Reader.reset(), which throws.
• Scale. No special setting; a one-character input is enough. Only inputs on which a length-changing mapping fires are affected (an empty input, or one the map never matches, is not), and whether a wrong value shows depends on the input, the read style, the assertion mode and the offset asked for. Over all 1,092 strings of length 1 to 6 on {a, b, ' '}, the re-read gave wrong offsets for 966 (a->xy), 602 (b->"") and 363 (ab->x) of them with assertions off.
• Effect. With assertions off, offsets too small for a longer replacement (a->xy) and too large for deleting or shrinking maps, often past the end of the input: over the same 1,092 strings, b->"" gives 516 too-large values inside the input and 774 past the end, ab->x 429 and 371, a->xy only too-small values (for example b->"" over "baa": correctOffset(1) = 3 instead of 2). With assertions on, the second read() instead throws BaseCharFilter's ordering AssertionError on 966 (b->""), 484 (ab->x) and 516 (a->xy) of those inputs. When the re-read completes, terms and matching are right.
• Likelihood. Rare: nothing in Lucene resets a char filter, and the recorded searches found no report since 2009.

new MappingCharFilter(map, new StringReader("a")) with the map a->xy is read to the end, rewound with reset(), read to the end again, and correctOffset(2) is compared with 1, the value a fresh filter gives. Deterministic, single thread; LuceneTestCase is only the base class. This input fails the same way with assertions on and off and does not hit the ordering assertion, so the failure is the wrong value itself.

Regression test (for the recommended fix): this same class (or its one method added to TestMappingCharFilter, next to testReaderReset); it passes with the recommended fix. Dropping reset() support needs a different test (expectThrows(IOException.class, f::reset)) and testReaderReset removed or rewritten; with the documentation-only option no offset-correctness test can pass.

cp $REPRO/TestMappingCharFilterResetOffsets.java $LUCENE/lucene/analysis/common/src/test/org/apache/lucene/analysis/charfilter/
cd $LUCENE && ./gradlew :lucene:analysis:common:test --tests TestMappingCharFilterResetOffsets -Ptests.seed=DEADBEEF -Ptests.asserts=true
# assertions off fails the same way: -Ptests.asserts=false

# gradle, stock 645d4fe14a20, -Ptests.seed=DEADBEEF -Ptests.asserts=true (stack trimmed; -Ptests.asserts=false identical)
   >     java.lang.AssertionError: correctOffset(2) after reset() expected:<1> but was:<0>
   >         at __randomizedtesting.SeedInfo.seed([DEADBEEF:6D266AAE1879C9E0]:0)
   >         at junit@4.13.2/org.junit.Assert.fail(Assert.java:89)
   >         at junit@4.13.2/org.junit.Assert.failNotEquals(Assert.java:835)
   >         at junit@4.13.2/org.junit.Assert.assertEquals(Assert.java:647)
   >         at org.apache.lucene.analysis.charfilter.TestMappingCharFilterResetOffsets.testCorrectOffsetAfterReset(TestMappingCharFilterResetOffsets.java:32)

:lucene:analysis:common:test (FAILURE): 1 test, 1  failure

# standalone runner, seed DEADBEEF: main 645d4fe14a20 and 10.6.0, -ea and -da, fail with the same message

• Deliberate support. MappingCharFilter.reset() (MappingCharFilter.java:62-68) was added by LUCENE-3830 (2012) to rewind the buffer, replacement and inputOff. LUCENE-1734 (2009) first made reset() work, because its reporter wanted "to call reset() to rewind reader in SOLR-1256", and added TestMappingCharFilter.testReaderReset (TestMappingCharFilter.java:65-79), which re-reads after reset() and checks characters only. Its input "x" matches none of the mappings in setUp() (TestMappingCharFilter.java:41-63), so it records no corrections and could not catch this bug even if it checked offsets.
• Legal Reader use. The java.io.Reader.reset() javadoc: "If the stream has not been marked, then attempt to reset it in some way appropriate to the particular stream, for example by repositioning it to its starting point. ... some support reset() without supporting mark()".
• Offsets are the point. CharFilter.java:22-25: "They can be used as java.io.Reader with additional offset correction"; MappingCharFilter.java:29-33: "correcting the resulting changes to the offsets".
• Lucene's own precondition breaks. BaseCharFilter.addOffCorrectMap documents and asserts non-decreasing offsets (BaseCharFilter.java:59-67, :77-87); MappingCharFilter violates it from a plain read() after reset().
• A committer called it supported. LUCENE-4321 (#5388, 2012), Robert Muir: "If someone (like MappingCharFilter) wants to support reset() and has tests for it (like MappingCharFilter), then thats fine" and "it supports reset() but not mark(), (only if also its input supports reset()) which is allowed". The outcome kept MappingCharFilter.reset().
• Nothing says otherwise. No javadoc, test, CHANGES or MIGRATE entry says offsets are invalid after reset().

The strongest case for "intended": CharFilter does not document reset(), markSupported() is false (inherited from Reader), and java.io.Reader.reset() says nothing about a subclass's offset metadata, so a maintainer can call the second-pass offset semantics underspecified. In the same LUCENE-4321 thread Uwe Schindler wrote that Lucene does not use those methods at all and that implementing them correctly with offsets is too heavy. The only test checks characters, on an input that no mapping matches.

Why it fails: the Reader contract only shows that calling reset() is legal, not what the offsets must be; the stronger evidence is the class's purpose (offset correction), its override that rewinds every other read coordinate, and its own tripped ordering assertion. The remark argues for not supporting reset() unless it is done right, not for supporting it with wrong offsets, and the thread then kept MappingCharFilter's reset() as supported (Robert Muir, above). Offsets past the input length and a tripped internal assertion are not a defensible behaviour. Upstream may fairly prefer to drop reset() support instead of fixing it; that is a choice of fix, not evidence that the behaviour is intended. The intended-behaviour devil's advocate and the adjudication rejected the argument, and the cross-model reviewers rated their own intended-behaviour argument weak.

One fix was validated in a scratch worktree of 645d4fe14a20; the alternatives are described but not run.

A protected clearOffCorrectMap() in BaseCharFilter, called at the end of MappingCharFilter.reset(). 2 files, +8 lines.

   // BaseCharFilter
+  /** Removes all recorded offset corrections, e.g. when a subclass rewinds its input in reset(). */
+  protected void clearOffCorrectMap() {
+    offsets = null;
+    diffs = null;
+    size = 0;
+  }

   // MappingCharFilter.reset()
     inputOff = 0;
+    clearOffCorrectMap();
   }

Trade-offs

• Adds a protected method to the public abstract BaseCharFilter; the name and shape are upstream's call. A package-private method would also work (both classes are in the same package), but subclasses in other modules could not use it. A downstream subclass may already declare a clearOffCorrectMap(): the call from reset() is virtual and would dispatch to it (or the subclass stops compiling if the signatures clash). final, a package-private method or another name avoids that; @lucene.internal only documents intent.
• A first read is unchanged; only code that calls reset() sees different offsets, now the same as a fresh filter's.
• A bare size = 0 (the approach of the unmerged 2010 LUCENE-2788 patch) is not enough on main: offsets stays non-null, so getLastCumulativeDiff() would read diffs[-1] (BaseCharFilter.java:55-57).
• Out of scope: if the underlying reader was mark()ed mid-stream before it was wrapped, input.reset() returns to the mark while inputOff restarts at 0.
• Not run: a MappingCharFilter over another MappingCharFilter. By reading the source, the fix handles it (the outer input.reset() resets the inner filter, which clears its own list, and CharFilter.correctOffset chains the two, CharFilter.java:71-77), and no other state stays stale (replacementPointer is unused once replacement is null; the buffer, the FST reader and scratchArc are reset or overwritten), but no executed test checks it.

What the checks validated

Gradle --no-daemon in a scratch worktree of 645d4fe14a20, OpenJDK 25.0.3 (2026-11-01). Stock: the repro fails with assertions on and off. With the fix:

• the repro passes with assertions on and off;
• org.apache.lucene.analysis.charfilter.*: 64 tests (TestHTMLStripCharFilter 36, TestHTMLStripCharFilterFactory 7, TestMappingCharFilter 18, TestMappingCharFilterFactory 2, the repro 1), 0 failures, seeds DEADBEEF and CAFEBABE;
• the whole :lucene:analysis:common:test, seed DEADBEEF: 340 test classes, 1,910 tests, 18 skipped, 0 failures (counted from the test-result XML);
• the other modules with BaseCharFilter subclasses compile (:lucene:analysis:nori, :lucene:analysis:icu);
• a probe with the fix gives the fresh filter's values after reset() for all three cases on this page ([0,0,1]; [1,2] for b->"" over "ba"; xy[0,1] b[2,3]), with assertions on and off;
• earlier, on main ea57360912ed: an independent 6-test JUnit (deleting and shrinking maps included) and the devil's advocate's 3-test tokenizer probe pass with the fix shadowed.

Remove the override, so reset() throws IOException("reset() not supported") from java.io.Reader, like every other CharFilter. Precedent: LUCENE-4072 (#5144), where reset() support was removed from ICUNormalizer2CharFilter before it was committed.

Trade-offs

Smallest change, but it removes behaviour that has worked for characters since 2009 and deletes testReaderReset; a breaking change, so main (11.0) with a MIGRATE note.

State that offsets are not valid after reset().

Trade-offs

Leaves the wrong offsets and the AssertionError inside read() in place.

Method: semi-formal (bounded-exhaustive translation validation, then an executed test).

• Finder. The hunt unit for analysis char filters, stemmers and tokenizers wrote a reference model of MappingCharFilter and checked the real filter against it for every map of 1 to 3 keys (keys of up to 3 characters over {a, b}, values "", x, xy, xyz) and every input of length up to 6 over {a, b, c}: output characters, correctOffset() at every position, two chunkings of the underlying reader and two read styles. The single-pass obligations held on 81,345,432 runs.
• Reset obligation. Read to the end, reset(), read again: same output and same offsets. At the smallest scope (one 1-character key, inputs over {a, b, c} of length at most 3) it failed 248 of the 640 reset checks with assertions off and 300 of 640 with them on (the ordering AssertionError); the scope has 960 runs, and the reset is checked on 2 of each input's 3 read styles. Every violation it printed was an offset mismatch, never an output mismatch.
• Confirmation. A public-API JUnit (a->xy over "a") failed on main with assertions on and off and on 9.12.3; a main() probe failed on releases back to 4.0.0.
• Hunt. Independent reproduction, cross-model reviews, an adjudication, the version-history runs, a prior-art search, a security screen, a credibility audit and a one-test minimal repro followed (see the checks below).