Stempel: Trie.reduce with Gener or Lift rewrites the trie it is given, even PolishAnalyzer's shared default table

The finding in 30 seconds

Stempel keeps its stemmer table as a trie of rows and cells. Lift and Gener shrink a trie by rewriting cells, and they do it on the input's own rows before copying them into the result. So the input is changed too, and for the shared Polish table that change reaches every analyzer in the JVM.

Diagram of the bug: PolishAnalyzer.getDefaultTable().reduce(new Gener()) is meant to return a new, smaller trie and leave the shared table unchanged. Gener clears rare commands in the shared table's own cells, then copies the rows into the result, so the shared table changes too. Every PolishAnalyzer in the JVM then stems 'książkami rzekami pracowali' to [książk, rzekami, pracowali] instead of [książek, rzekść, pracować]. Lift does the same on a trie that is not lifted yet; Optimizer and Optimizer2 leave their input unchanged. — The shared Polish table before and after `getDefaultTable().reduce(new Gener())`: the reduction rewrites the table's own cells before it copies them, so a new `PolishAnalyzer` stems 'książkami' to 'książk'. Select the still to open it at full size. Generated with AI, checked against the code.

You want a smaller Polish stemmer table and call PolishAnalyzer.getDefaultTable().reduce(new Gener()). The javadoc calls that table "unmodifiable", and reduce returns "the newly reduced Trie".
You expect the shared table, and so every other PolishAnalyzer, to stay as it was: "książkami rzekami pracowali" still stems to [książek, rzekść, pracować].
Instead Gener clears rare commands in the shared table's own cells, 42,134 of its 121,822, before it copies the rows into the new trie. 'książkami' loses the last piece of its command, Ie (insert e): that piece had count 2 in a row whose threshold is 4, a tenth of its total 43.
From then on every new PolishAnalyzer gives [książk, rzekami, pracowali], with no error, for as long as the class stays loaded (normally until the JVM exits). A document indexed before the call with 'książkami' is no longer found by the query 'książkami': 1 hit before, 0 after.
Lift does the same to a trie that is not lifted yet: after reduce(new Lift(false)) of a trie that holds only aa -> Da, the input itself answers Da for the word 'a'. Optimizer and Optimizer2 leave their input alone.

Who is affected

Only applications whose own code calls reduce(new Gener()) or reduce(new Lift(..)) on a trie they keep using, for example on PolishAnalyzer.getDefaultTable(): then every PolishAnalyzer and stempelPolishStem factory in the JVM stems differently. Normal analysis, the factory and the Compile tool never do this, and no public caller of getDefaultTable().reduce was found.

Mechanism, terms and numbers654 words

Trie.reduce(Reduce) is documented to "return the newly reduced Trie", and PolishAnalyzer.getDefaultTable() "Returns an unmodifiable instance of the default stemmer table". But Lift.optimize and Gener.optimize work on the input's own list of rows (List<Row> orows = orig.rows;) and rewrite its cells in place (Lift moves a child row's command up into the parent cell and cuts the link; Gener drops infrequent commands, and links to rows left empty) before they copy the rows into the returned trie. The input trie answers differently afterwards. For the shared default table this is visible in the whole JVM: after getDefaultTable().reduce(new Gener()) every new PolishAnalyzer, and every stempelPolishStem factory stream, stems "książkami rzekami pracowali" to [książk, rzekami, pracowali] instead of [książek, rzekść, pracować]. Optimizer and Optimizer2 leave their input unchanged. Only application code that calls reduce triggers it; every release from 3.1.0 (the default-table form from 4.0.0) to 10.6.0 (10.6.0 by reading) and main is affected. Copying the rows first fixes it; upstream TestStemmer, which reduces one trie five times in a row, then needs one entry made explicit.

Terms

Trie, row, cell. Stempel's table is a trie: one row per node, and in each row one cell per next character. A cell holds cmd (an index into the list of patch commands that turn a word into its stem), ref (the child row, or -1), skip and cnt (a frequency count). Stemming walks the word through the trie with getLastOnPath and applies the last command it finds.
Patch command. Read from the end of the word: Dc-aIe is delete 3 letters, skip 1, insert e, so książkami -> książek.
Reduction. trie.reduce(r) calls r.optimize(trie). Optimizer and Optimizer2 merge rows; Lift moves a command shared by a whole child row up into the parent cell; Gener drops commands that are rare in their row.

The bug

Lift.optimize takes the input's own list, List<Row> orows = orig.rows; (Lift.java:92), and calls liftUp on every row. liftUp writes skip, cnt, cmd and ref = -1 into those cells (Lift.java:114-146).
Gener.optimize does the same (Gener.java:81): eat sets ref = -1 on a link to a row that became empty and clears cmd and cnt of a cell whose count is under a tenth of its row's total (Gener.java:105-129).
Only then does Reduce.removeGaps copy each reachable row into a new Remap row with new Cell objects (Reduce.java:88-128). The result shares nothing with the input; the input keeps the rewritten cells.
Optimizer builds its new rows before it changes anything (Optimizer.java:86), so it never writes its input.
PolishAnalyzer.getDefaultTable() returns one static MultiTrie2, loaded once; every PolishAnalyzer and StempelPolishStemFilterFactory stems with it (PolishAnalyzer.java:60-63, :121; StempelPolishStemFilterFactory.java:50). MultiTrie2.reduce reduces every sub-trie (MultiTrie2.java:279), so all of them are rewritten.

The numbers

what you do	expected	actual
`getDefaultTable().reduce(new Gener())`, then a new `PolishAnalyzer` on "książkami rzekami pracowali"	[książek, rzekść, pracować]	[książk, rzekami, pracowali]
the same on a private loaded copy of the bundled table	copy unchanged	14,075 of 67,668 probed answers change (another probe: 3,352 of 20,012 stemmed words)
trie {aa -> Da}, `reduce(new Lift(false))`, then `getLastOnPath("a")` on the input	nothing	`Da`
the default table's cells, before and after that call	unchanged	42,134 of 121,822 cells lose their command; for 'książkami' the piece `Ie` (count 2) in a row with total 43 (threshold 4)
a document indexed before the call with "książkami", queried for "książkami" after it	1 hit	0 hits (the query is now `książk`, the document holds `książek`)
control: `reduce(new Optimizer())` or `reduce(new Optimizer2())` on a loaded copy	copy unchanged	unchanged

The shipped table was compiled with the method string -0ME2, which already applies Lift(false), so Lift on it changes 0 stems; only Gener shows there. The Lift half shows on a trie that is not lifted yet, such as a custom table being built.

Exposure and likelihood270 words

Who is hit. Applications that call reduce with Gener or Lift on a trie they keep using. Two plausible shapes: reducing getDefaultTable() to get a smaller table for a custom StempelStemmer, which silently changes every Polish analyzer in the JVM; or trying several reductions on one table being built, as Lucene's own TestStemmer does, where each reduction runs on what the previous one left.
Not the default path. Stemming only reads the table (getLastOnPath), and Compile always does trie = trie.reduce(..) and drops its input. A public code search found no getDefaultTable().reduce caller and only copies of TestStemmer reducing with Gener or Lift. No prior report of it was found.
What goes wrong. Silent wrong stems, no error. Index-time and query-time analysis disagree: affected terms indexed before the call stop matching queries analysed after it, and documents indexed after it get different terms. Analyzers created before the call hold the same table, so they change too (code reading). The shared table stays changed for as long as the PolishAnalyzer class stays loaded, normally until the JVM exits (a static final loaded once).
Scale. Deterministic in every executed case (one-key tries, the exhaustive small tries, the bundled table), with assertions on and off. On a loaded copy of the bundled table Gener changed the stems of 193,552 of 366,667 probe words in one probe, 14,075 of 67,668 answers in another and 3,352 of 20,012 stemmed words in a third.
Concurrency (not executed). Analysis on another thread during a reduce call reads cell fields while they are rewritten, without synchronization; by code reading that gives wrong stems there.

Minimal repro

The test shows that Trie.reduce(new Lift(false)) on a one-key MultiTrie2 {aa -> Da} built by the caller rewrites the input trie in place (Lift.optimize works on orig.rows), although reduce is documented to return "the newly reduced Trie"; Gener has the same root cause.

TestTrieReduceKeepsInput.java (JUnit test)32 lines

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package org.apache.lucene.analysis.stempel;

import org.apache.lucene.tests.util.LuceneTestCase;
import org.egothor.stemmer.Lift;
import org.egothor.stemmer.MultiTrie2;

public class TestTrieReduceKeepsInput extends LuceneTestCase {

  public void testLiftDoesNotModifyItsInput() {
    MultiTrie2 trie = new MultiTrie2(true);
    trie.add("aa", "Da");
    trie.reduce(new Lift(false));
    // reduce returns a new Trie, so the input should still have no answer for "a"; actual: "Da"
    assertEquals("", trie.getLastOnPath("a").toString());
  }
}

Command

# in a copy of this finding's repro folder (repro/LFV-696/, which holds src/)
# Lucene checkout at the tested commit, built with: ./gradlew --no-daemon assemble   (JDK 25)
LUCENE=/path/to/lucene; V=12.0.0-SNAPSHOT
CP=$LUCENE/lucene/core/build/libs/lucene-core-$V.jar
CP=$CP:$LUCENE/lucene/codecs/build/libs/lucene-codecs-$V.jar
CP=$CP:$LUCENE/lucene/test-framework/build/libs/lucene-test-framework-$V.jar
CP=$CP:$LUCENE/lucene/analysis/common/build/libs/lucene-analysis-common-$V.jar
CP=$CP:$LUCENE/lucene/analysis/stempel/build/libs/lucene-analysis-stempel-$V.jar
CP=$CP:randomizedtesting-runner-2.9.2.jar:junit-4.13.2.jar:hamcrest-3.0.jar   # test-framework dependencies (Maven Central)
CP=$CP:junit-platform-console-standalone-6.1.3.jar   # JUnit Platform console launcher (Maven Central)

javac -d classes -cp "$CP" $(find src -name '*.java')

# run 1: assertions on, seed DEADBEEF
java -ea -esa --add-modules jdk.incubator.vector --enable-native-access=ALL-UNNAMED -Dtests.seed=DEADBEEF \
     -cp "classes:$CP" org.junit.platform.console.ConsoleLauncher \
     execute --select-class org.apache.lucene.analysis.stempel.TestTrieReduceKeepsInput

Expected vs actual: Expected: the input still answers getLastOnPath("a") = "" after the reduction. Actual: "Da" (expected:<[]> but was:<[Da]>, with -ea and -da).

All files of this repro: repro/LFV-696/

More files, environment, full output32 words

Environment

lucene	`Lucene main e1faeba08bde (2026-11-02)`
jdk	`25.0.3+9-LTS`

Observed output

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: expected:<[]> but was:<[Da]>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.analysis.stempel.TestTrieReduceKeepsInput.testLiftDoesNotModifyItsInput(TestTrieReduceKeepsInput.java:30)

Why it looks unintended

getDefaultTable() "Returns an unmodifiable instance of the default stemmer table"; its CHANGES entry (SOLR-3737) says "ensure immutability".
Trie.reduce returns "the newly reduced Trie"; the sibling reductions Optimizer and Optimizer2 leave their input unchanged.
Upstream TestStemmer reduces one trie five times, then re-checks the original, as if its author expected the input to stay intact (the test does not say so).

The full argument230 words

The contract. Trie.reduce: "Remove empty rows from the given Trie and return the newly reduced Trie. ... @return the newly reduced Trie" (Trie.java:361-369). Gener.optimize: "@return a new optimized Trie" (Gener.java:75). PolishAnalyzer.getDefaultTable(): "Returns an unmodifiable instance of the default stemmer table." (PolishAnalyzer.java:60).
Lucene's own intent for the table. SOLR-3737 (04f3188bf9b, 2012) added getDefaultTable() with that javadoc, made the factory share the one table and, in the same commit, made Trie.add package-private. CHANGES: "Also, ensure immutability and use only one instance of this table in RAM (lazy loaded) since it's quite large." (lucene/CHANGES.txt:15072-15074). reduce() is still a public way to change it.
The siblings. Optimizer and Optimizer2 copy before they change anything; three independent controls show their input unchanged. Writing into the input is not the reduction family's design.
The upstream test. TestStemmer.assertTrieContents (from the 2010 import) builds {trie, reduce(Optimizer), reduce(Optimizer2), reduce(Gener), reduce(Lift(true)), reduce(Lift(false))} from one trie and checks every entry, the original included. It passes today only because its Lift(false) entry runs on the trie that Lift(true) already rewrote: on a fresh trie reduce(new Lift(false)).getFully("ba") is null. It does check the original trie, for the inserted keys only, which this rewrite happens not to break.
Public API. org.egothor.stemmer is exported by the module, with no internal or experimental marker; the repro needs no cast and no reflection.

The devil's-advocate check238 words

The strongest case for "intended" (the adjudication rated it "moderate at most"; the cross-model intended-lens review rated it moderate, and the devil's advocate found a moderate reading only for caller-built tries): Lift and Gener are legacy egothor compiler passes; Compile always rebinds trie = trie.reduce(..) and discards the input; Trie.reduce's first sentence, "Remove empty rows from the given Trie", can be read as working on the given trie; Lift.optimize's own javadoc says only "return the reduced Trie"; liftUp and eat are public helpers that work in place (liftUp's javadoc describes propagating commands up the trie; eat's calls it a test of whether a row should stay, yet it clears ref, cnt and cmd); every reducer's result shares the input's command list, so reduce never promised a fully independent copy; and "unmodifiable" was never deeply enforced, because the default table is a MultiTrie2 whose add is still public, reachable with a downcast.

Why it fails: for the default table the promise is Lucene's own text (javadoc and CHANGES), not inherited egothor wording; reduce needs no cast, is a method of the declared Trie type, says "newly reduced" and names no side effect; no reducer ever writes the shared command list; the optimizers keep their input. For a trie the caller built, the argument survives in weaker form, and a maintainer may prefer a javadoc note for that half. Verdict: a defect.

Suggested fix

Recommended Copy the rows in Lift.optimize and Gener.optimize

Copy the rows before Lift and Gener rewrite them: a small Reduce.copyRows helper and one line in each reduction (+20/-2). Upstream TestStemmer then needs its Lift(false) entry made explicit as reduce(Lift(true)).reduce(Lift(false)), which is what it always tested. Validated: the repro passes, the whole stempel module passes (21 tests) and check -x test passes (format, ecjLint, forbiddenApis).

static List<Row> copyRows(List<Row> rows) {   // in Reduce
  List<Row> copy = new ArrayList<>(rows.size());
  for (Row row : rows) {
    Row r = new Row();
    for (CharCursor ch : row.cells.keys()) {
      r.cells.put(ch.value, new Cell(row.at(ch.value)));
    }
    copy.add(r);
  }
  return copy;
}
// Lift.optimize and Gener.optimize:
-    List<Row> orows = orig.rows;
+    List<Row> orows = copyRows(orig.rows);

All options and trade-offs4 options · 623 words

One fix was built and validated with gradle in a clean scratch worktree; the alternatives are sketches.

Copy the rows in Lift.optimize and Gener.optimize (validated by the checks) (recommended)

Add a package-private Reduce.copyRows(List<Row>) that copies every row and cell, and use it in Lift.optimize and Gener.optimize instead of orig.rows. liftUp and eat stay what they are, in-place steps on the rows they are given. Update TestStemmer in the same change (below). The snippet is condensed; the full patch is fix-validation/fix.diff (+20/-2), and the test update is fix-validation/teststemmer.diff.

static List<Row> copyRows(List<Row> rows) {   // in Reduce
  List<Row> copy = new ArrayList<>(rows.size());
  for (Row row : rows) {
    Row r = new Row();
    for (CharCursor ch : row.cells.keys()) {
      r.cells.put(ch.value, new Cell(row.at(ch.value)));
    }
    copy.add(r);
  }
  return copy;
}
// Lift.optimize and Gener.optimize:
-    List<Row> orows = orig.rows;
+    List<Row> orows = copyRows(orig.rows);

Trade-offs

Not test-neutral. With the fix alone, upstream TestStemmer fails 5 of 6, on getFully for its Lift(false) entry. That entry has always tested Lift(false) after Lift(true), because Lift(true) had already rewritten the test's own trie. On a fresh trie Lift(false) folds a uniform child row into its parent cell with skip = 0, so getFully("ba") is null while getLastOnPath("ba"), which the stemmer uses, is still 2. The test update keeps the existing assertion statements but changes what the sixth entry means: it is now written as the chain it has always run, trie.reduce(new Lift(true)).reduce(new Lift(false)); a new loop checks Lift(false) on the unchanged trie with getLastOnPath only, not getFully. Whether getFully after Lift(false) should keep the longer keys is a separate question; open issue #6584 may touch it (code reading).
Cost. One extra copy of every row per Lift or Gener reduction; the bundled table has 6,815 rows over 8 sub-tries. Not benchmarked; reductions run when a table is built, not during analysis.
No API change. A trie reduced once gets the same answers as before, because the reduction runs on an identical copy (code reading).

What the checks validated

Gradle (--no-daemon) in a clean scratch worktree of a9779ef3c193, JDK 25, seed DEADBEEF, removed afterwards:

stock: the 2 new tests fail with assertions on and off; TestStemmer passes 6 of 6; the whole module runs 21 tests, 2 fail (the new ones).
fix only: the 2 new tests pass; TestStemmer fails 5 of 6 (all but testMultiTrie2), with assertions on and off.
fix + TestStemmer update: 8 of 8 with assertions on and off; the whole :lucene:analysis:stempel:test passes, 6 classes and 21 tests (TestPolishAnalyzer, TestPolishSnowballAnalyzer, TestStempelPolishStemFilterFactory, TestCompile, TestStemmer, TestReduceKeepsInput); :lucene:analysis:stempel:check -x test passes (checkGoogleJavaFormat, ecjLint, forbiddenApis; error-prone and eclint did not run in that build).
An equivalent inline copy (the finder's patch) also passed the independent reproducer's 9 tests, a cross-model reviewer's 5 and the devil's advocate's 2 expectation tests.

Copy inside Trie.reduce

Have Trie.reduce hand a deep copy of itself to by.optimize(..).

Trade-offs

Covers Reduce subclasses written by users too, but Reduce.optimize is public and can be called directly, bypassing Trie.reduce, and the copy is wasted for Optimizer and Optimizer2. Needs the same TestStemmer update. Not run.

Protect only the shared default table

Make the default table copy itself before any reduction.

Trade-offs

Settles the "unmodifiable" contract with the smallest change, but leaves tries the caller built rewritten in place. Not run.

Document that Lift and Gener modify their input

State it in the javadoc of Trie.reduce, Lift and Gener, and say in getDefaultTable() that it must not be reduced.

Trade-offs

Cheapest, but "unmodifiable" then holds only by convention. A maintainer may prefer this for caller-built tries, combined with the previous option.

Downstream note

Downstream, every index in a JVM that analyses Polish with PolishAnalyzer or the stempelPolishStem factory reads this one table, so one component's reduce(new Gener()) call changes stemming for all of them.

How it was found

Bounded-exhaustive testing of the stempel reductions: a harness queried each input trie again after reducing it, over 183,540 small tries (1 to 4 keys of up to 3 letters on {a, b}, three fixed commands, both directions). Lift changed its input in most, Optimizer and Optimizer2 never; a bundled-table probe showed Gener. A public-API JUnit reproduced it.

The method in full189 words

Method: bounded-exhaustive translation validation (the semi-formal tier), in a hunt unit on the stempel trie code (diff, store/load round trips and reductions).

The harness built every set of 1 to 4 keys of length at most 3 on the alphabet {a, b}, each key given one of three fixed commands (Da, Db, -aDa), in both directions (183,540 tries), applied single reductions and pairs of them, and compared the answers with a reference; one column queried the input trie again after each single reduction.
That column listed input changes only for Lift(false) (168,462 tries) and Lift(true) (160,590); Optimizer and Optimizer2 never changed their input. Gener drops nothing in tries that small (its threshold is a tenth of a row's total count).
A companion probe on a loaded copy of the bundled table found that reduce(new Gener()) changes the stems of 193,552 of 366,667 probe words on that copy, while Lift changes none (the table is already lifted).
The hypothesis came while modelling the reduction step from the code; the harness witnessed it, and it was reduced to the 2-test public-API JUnit on the stock main jar.

Prior reports, history and checks

Prior reports

No prior report found (2026-10-27): 76 recorded tool queries over GitHub issues, PRs and commits, downstream repositories, the mailing lists, JIRA and the web, manual GitHub searches and a scan of all 409 open PRs. Closest: #6584 (LUCENE-5521, open), a different Compile failure; #8841 and SOLR-3737 are context.

Searches5 searches

Verdict: new; not fixed upstream. No issue, PR, commit, JIRA ticket, list thread or downstream report says that reduce with Lift or Gener changes its input, or that the default table can be changed this way.

What was searched, when, and what turned up
Where	When	How	Found
apache/lucene issues, PRs, commits; downstream repositories; dev@, java-user@, issues@, commits@; JIRA archive; web	2026-10-27	scripted prior-report search with a spec for this bug: 24 GitHub, 4 commit, 4 downstream, 32 list, 6 JIRA and 6 web queries (492 distinct hits, 0 high); plus the hunt unit's run (6,891 hits, 0 high)	no report
GitHub, manual	2026-10-27	18 searches (egothor, stempel issues and PRs, PolishAnalyzer, StempelStemmer, Trie.reduce, DEFAULT_TABLE, ...); the changed files of all 409 open apache/lucene PRs; an independent reviewer's own searches	only #681 (lint) and #16911 (branch_10x) touch stempel files, neither the fix site
Downstream and forks	2026-10-27	Elasticsearch, OpenSearch, Solr, the Lucene.NET port, two Egothor forks	no report; the Lucene.NET port has the same in-place code
JIRA and lists	2026-10-27	LUCENE (stempel 30, egothor 11), SOLR (stempel 14; SOLR-3737 read in full), LUCENENET; java-user@, dev@, solr-user@, issues@	no report; SOLR-3737 is the origin of the 'unmodifiable' contract and never mentions reduce
Related items (read)	2026-10-27	#6584 (LUCENE-5521), #8841 (LUCENE-7790), #6569, #7678	#6584: a TestCompile failure on Compile's output, where the input is discarded, a different defect; #8841: a maintainer prefers not to modify the forked code

History

In Lucene since the stempel import (98c47c57e0a, LUCENE-2298, 2010; first release 3.1.0). SOLR-3737 (04f3188bf9b, 2012; first in 4.0.0) made the table a public JVM-wide singleton. A portable probe fails on all 138 releases from 3.1.0 to 10.5.2 and on main; not fixed upstream.

Versions tested6 rows

Introduced by the stempel import itself: 98c47c57e0a "LUCENE-2298: Add stempel, an algorithmic stemmer with included Polish support" (committed by @rmuir on 2010-05-03; contributed code, CHANGES credits Andrzej Bialecki; JIRA LUCENE-2298, now #3374; no pull request), backported to branch_3x as 58328f03790 and first released in 3.1.0. The import already had orows = orig.rows with liftUp and eat writing the input's cells; later commits to Lift.java and Gener.java only reformat or retype them (for example #13420, #13422, #14832, #14965).

JVM-wide since SOLR-3737 (04f3188bf9b on trunk, b3818670d35 on branch_4x, 2012-08-17), which added the public getDefaultTable() and made the factory share the one table, without touching reduce, Lift or Gener. First release 4.0.0. Not a regression.

Versions and branches executed, with results
Version or commit	Verdict	Result
main `a9779ef3c193`	affected	the JUnit fails 2 of 2 (gradle and classpath runner, `-ea` and `-da`); the fix passes
10.5.2, 10.0.0, 9.12.3, 9.1.0	affected	the JUnit fails 2 of 2 (10.5.2 also with `-da`)
4.0.0 to 10.5.2, every release (130)	portable probe	`Lift` and `Gener` change a caller-owned trie, and reducing `getDefaultTable()` changes a new `PolishAnalyzer`'s terms; the `Optimizer` control never changes
3.1.0 to 3.6.2 (8)	no getDefaultTable	a caller-owned trie changes; the default table is not public yet ('API absent')
3.0.3	no module	no stempel module; the probe does not compile
upstream main `c5ab3e57b9ba`, `branch_10x`, `branch_10_6`	source only	the same lines; not executed

Independent checks

An independent reproducer, two executed cross-model reviews (OpenAI GPT-5.6), an intended-behaviour devil's advocate and an adjudication all found it real; a credibility audit re-ran it; this doc's gradle runs reproduce it and validate the fix.

Checks and verdicts304 words

Every verdict below rests on executed runs. The adjudication corrected one point of the first write-up: the returned trie is a copy and shares no row or cell with the input; the defect is the rewrite before that copy.

Each independent check and its verdict
Check	Verdict	What it found
Cross-model review, refute lens (OpenAI GPT-5.6, executed)	real (high)	Reproduced both failures; root cause agreed; discarded a byte-comparison oracle because hash-map iteration makes `store()` bytes unstable and relied on lookups and stems only; `fix_fixes_it` false only because `TestStemmer` needs the update. Ran on main `ea57360912ed`, whose stempel sources equal `a9779ef3c193`'s.
Independent reproducer	real	Own 9-test JUnit fails 5 of 9 on main (`-ea`, `-da`), 10.5.2 and 9.12.3; `Optimizer`/`Optimizer2` controls pass; a side check showed the `TestStemmer` dependency on the rewrite.
Cross-model review, intended lens (the cross-model review, OpenAI GPT-5.6, executed)	real (high)	Intended-behaviour argument moderate; own test fails 2 of 5 (controls pass) and passes 5 of 5 with the fix; the review ran on main `ea57360912ed` (same stempel sources) and a separate run re-ran its test on `a9779ef3c193`.
Intended-behaviour devil's advocate	real (medium)	No intended reading survives for the default table; a moderate design-history reading survives for caller-built tries. Own probe fails 2 of 7 on main, 10.5.2 and 9.12.3; 5 controls pass.
Adjudication	confirmed (high)	Re-ran the independent reproduction's test (5 of 9 fail, `-ea` and `-da`) and with the fix (9 of 9 pass); corrected the 'same Row objects' wording.
Credibility audit	keep	Re-ran the JUnit with `-da` (2 of 2 fail); score and class hold.
This doc's gradle runs	reproduced; fix validated	Stock: 2 of 2 fail, assertions on and off; fix + `TestStemmer` update: the whole module passes and `check -x test` passes.

Diagram file

The diagram as a still image, generated with AI and checked against the code. It is under 10 MB, the limit for a GitHub attachment.

PNG 572 KB

Sources and provenance

Finding record, Lucene and upstream sources, repro files and diagram notes1,775 words

Finding record

LFV-696, snapshot of 2026-11-06

Registry title: Stempel's Trie.reduce with Gener or Lift rewrites the cells of the trie it is given, so reducing PolishAnalyzer.getDefaultTable(), documented as unmodifiable, silently changes the stems of every later PolishAnalyzer and StempelPolishStemFilterFactory in the JVM
Area: analysis/stempel / egothor stemmer trie reductions (Trie.reduce with Lift and Gener; the shared PolishAnalyzer.getDefaultTable() table)
Priority: Low, rank 504: Once triggered every Polish stemmer in the JVM silently changes its stems, but only when application code calls Trie.reduce with Gener or Lift on a table it keeps using, which normal analysis, the factory and Compile never do.
Impact class: silent wrong results
Impact note: Hits only applications whose own code calls Trie.reduce (or MultiTrie/MultiTrie2.reduce) with Gener or Lift on a trie it keeps using, most visibly PolishAnalyzer.getDefaultTable(): after getDefaultTable().reduce(new Gener()) every PolishAnalyzer and StempelPolishStemFilterFactory in the JVM stems differently (on a loaded copy of the bundled table 14,075 of 67,668 probe answers change in the independent reproduction's test, and 'książkami' becomes 'książk' instead of 'książek'), so terms indexed before the call stop matching queries analysed after it, deterministically with -ea and -da, at any size, in every release from 3.1.0 (the default-table form from 4.0.0) to 10.6.0 (10.6.0 by reading) and on main; normal analysis, the factory and the Compile tool never reduce a table they keep using, and on the shipped table only Gener changes answers (it is already lifted), so the Lift half shows only on a trie that has not been lifted yet, such as a custom table being built.
Summary: You call reduce(new Gener()) or reduce(new Lift(..)) on a stempel Trie, for example on PolishAnalyzer.getDefaultTable(), which is documented as 'an unmodifiable instance of the default stemmer table', and expect what Trie.reduce's javadoc says, 'the newly reduced Trie', with the input left as it was (Optimizer and Optimizer2 behave so). Instead Lift.optimize and Gener.optimize take the input's own row list and rewrite its cells in place (Lift moves commands up a level and clears references, Gener drops infrequent commands and references) before copying them into the result, so the input answers differently afterwards: after PolishAnalyzer.getDefaultTable().reduce(new Gener()) every new PolishAnalyzer stems 'książkami rzekami pracowali' to [książk, rzekami, pracowali] instead of [książek, rzekść, pracować], and a one-key trie {aa -> Da} reduced with Lift(false) starts to answer getLastOnPath('a') = Da instead of nothing. Copying the rows before the rewrite fixes it; upstream TestStemmer, which reduces one trie five times, then needs a fresh trie per reduction.
Introduced: 98c47c57e0a54ab4aab9d91dd60c33edef130893 (trunk, SVN r940433, 'LUCENE-2298: Add stempel, an algorithmic stemmer with included Polish support', JIRA LUCENE-2298 'Polish Analyzer', now GitHub issue #3374; a JIRA patch by @rmuir of contributed code, CHANGES credits Andrzej Bialecki via Robert Muir; SVN era, no pull request), branch_3x backport 58328f0379049045b325d2082295836a7a50e9ce ('LUCENE-2298: backport', r941704, the copy that shipped in 3.1.0): the import already had Lift.optimize take 'orows = orig.rows' and liftUp rewrite those cells, and Gener.optimize/eat clear cells of the input in place [read: git show of the commit; blame at the study base 7cecf795236, whose stempel sources are identical to main; later commits to Lift.java and Gener.java only reformat or retype, e.g. #13420/#13422 (HPPC map), #14832, #14965]. The JVM-wide default-table consequence was enabled by 04f3188bf9b8ce059f72399bad721d9190ed2dc0 (trunk) / b3818670d3545bfc9abae3bcdf766da4ca7308c0 (branch_4x), SOLR-3737 by @rmuir, which added the public PolishAnalyzer.getDefaultTable() ('Returns an unmodifiable instance of the default stemmer table'), made StempelPolishStemFilterFactory share that one table and made Trie.add package-private, but did not touch reduce, Lift or Gener [read]. Not a regression: the in-place rewrite is as old as the module in Lucene, and the Lucene.NET port has the same code [read]. (2010-05-03 (trunk 98c47c57e0a); branch_3x backport 2010-05-06 (58328f03790); default-table consequence 2012-08-17 (SOLR-3737, 04f3188bf9b / b3818670d35))
First affected version: 3.1.0 [exec: the version-portable history probe (main() mode, public API, getDefaultTable looked up by reflection) shows Lift and Gener changing a caller-owned trie on 3.1.0 while its Optimizer control leaves the input unchanged; 3.1.0 is the first release with the stempel module]; 3.0.3 has no stempel API (the probe does not compile) [exec], so no release has the API without the bug. The default-table form first in 4.0.0 [exec: 'API absent' on all eight 3.x releases, fails on 4.0.0; 4.0.0-ALPHA and -BETA not run, and the BETA tag predates the SOLR-3737 commit, read].
Affected versions: every release from 3.1.0 through 10.5.2 and main a9779ef3c193 (unreleased 11.0) [exec: the portable probe fails on all 138 releases from 3.1.0 to 10.5.2 (-ea) and on main, Lift and Gener change a caller-owned trie in each and the Optimizer control never changes; its default-table check is 'API absent' on the eight 3.x releases and fails on all 130 releases from 4.0.0 to 10.5.2 and on main; the finder's stock JUnit fails 2 of 2 on 9.1.0, 9.12.3 and 10.0.0 (-ea) and on 10.5.2 and main (-ea and -da), and does not compile on 9.0.0 (test-framework packages renamed in 9.1.0); the independent reproduction's test fails 5 of 9 on main (-ea and -da), 10.5.2 and 9.12.3 (-ea); the intended-behaviour check's probe fails 2 of 7 on main (-ea and -da), 10.5.2 and 9.12.3 (-da)]. branch_10x and branch_10_6 have the same lines [read], so 10.6.0 (released 2026-10-29 from branch_10_6) is affected too. Not fixed upstream: upstream main a95c754ee003 (2026-10-26) has the same Lift/Gener lines, no commit after a9779ef3c193 touches lucene/analysis/stempel, and no open PR touches the fix site (2026-10-27) [read]. lucene/analysis/stempel is identical at ea57360912ed and a9779ef3c193 (empty git diff). Executed on Linux aarch64 with JDK 25 only.
Status: documented (brief, checks and a per-finding doc exist)
Verification: Confirmed by AI-run checks: an independent repro, a cross-model review that agreed, an intended-behaviour check, a prior-report search and a skeptical audit; not yet triaged by a person.
Found by: semi-formal: exhaustive check
Upstream and prior reports: none found (2026-10-27, read-only; the prior-report record verdict new): a scripted prior-report search with a candidate-specific spec (24 GitHub issue/PR queries on apache/lucene, 4 commit, 4 downstream-repository, 32 mailing-list over dev@, java-user@, issues@ and commits@, 6 JIRA-archive queries and the web; 492 distinct hits, 0 high) plus the unit-wide run (6,891 hits, 0 high); manual GitHub searches of apache/lucene (egothor 11, stempel issues 27 and PRs 6, PolishAnalyzer 4, StempelStemmer 2, 'Trie.reduce' 0, DEFAULT_TABLE 0, and others, all read by title); the file lists of all 409 open apache/lucene PRs (only #681, a lint change, and #16911, the branch_10x decommission, touch stempel files, neither the fix site); elastic/elasticsearch, OpenSearch, Solr, the Lucene.NET port and the Egothor forks buda-base/stemmer and Radixor; JIRA LUCENE (stempel 30, egothor 11), SOLR (stempel 14; SOLR-3737 read in full) and LUCENENET; the java-user@, dev@, solr-user@ and issues@ lists; and the full git history (no commit changes Lift or Gener behaviour since 2010). The intended-behaviour check ran its own GraphQL searches and found nothing either. Nearest item, not the same bug: #6584 (LUCENE-5521, open since 2014, 'Egothor Stemmer Bug for Optimizing'), a TestCompile failure on Compile's reduced output once Compile honours upper-case method letters; Compile discards the input, so the in-place rewrite plays no role there (it may share only the getFully-after-Lift(false) behaviour that the fix exposes in TestStemmer, by code reading, not executed). Context only: #8841 (LUCENE-7790, Trie.add visibility, with a maintainer comment against modifying forked code), SOLR-3737 (origin of the 'unmodifiable' contract), #6569 and #7678 (ignored toUpperCase in Compile), buda-base/stemmer#1 (reductions are lossy by design, a different property). Not fixed upstream. Gaps: two web queries throttled; GitHub search reads only the first 50 results per query, which affects only the broad 'reduce' queries.
Lucene commit tested: main a9779ef3c193: gradle in a clean scratch worktree (assertions on and off, 2026-10-28) and the hunt's classpath runner (-ea and -da); the same JUnit on 10.5.2, 10.0.0, 9.12.3 and 9.1.0; a version-portable main() probe on all 138 releases from 3.1.0 to 10.5.2 (-ea). Upstream main c5ab3e57b9ba (2026-10-28) has no stempel commit after a9779ef3c193 (read only).

Lucene and upstream sources

Finding record: lucene/analysis/stempel/src/java/org/egothor/stemmer/Lift.java:82-103 (at a9779ef3c193) (optimize, javadoc 'Optimize (eliminate rows with no content) the given Trie and return the reduced Trie'; 'List<Row> orows = orig.rows;' at :92; liftUp(orows.get(j), orows) for every row, then removeGaps), :114-145 (liftUp writes c.skip, c.cnt, c.cmd and c.ref of the input's cells, :118-142)
lucene/analysis/stempel/src/java/org/egothor/stemmer/Gener.java:71-96 (at a9779ef3c193) (optimize, javadoc 'Return a Trie with infrequent values occurring in the given Trie removed', '@return a new optimized Trie'; orows = orig.rows at :81), :105-129 (eat: c.ref = -1 at :112; c.cnt = 0 and c.cmd = -1 at :121-122)
lucene/analysis/stempel/src/java/org/egothor/stemmer/Trie.java:361-369 (at a9779ef3c193) (reduce, 'Remove empty rows from the given Trie and return the newly reduced Trie.', '@return the newly reduced Trie', body 'return by.optimize(this)'); MultiTrie.java:176-189 and MultiTrie2.java:271-284 (reduce every sub-trie, same javadoc)
lucene/analysis/stempel/src/java/org/egothor/stemmer/Reduce.java:88-128 (at a9779ef3c193) (removeGaps replaces each row with new Remap(now, remap), whose cells are new Cell copies, so the returned trie shares no Row or Cell with the input); Optimizer.java:84-88 ('Row now = new Remap(orows.get(j), remap)' before merging: Optimizer never writes its input)
lucene/analysis/stempel/src/java/org/apache/lucene/analysis/pl/PolishAnalyzer.java:60-63 (at a9779ef3c193) ('Returns an unmodifiable instance of the default stemmer table.', returns DefaultsHolder.DEFAULT_TABLE); org/apache/lucene/analysis/stempel/StempelPolishStemFilterFactory.java:50 (create uses PolishAnalyzer.getDefaultTable()); module-info.java:23-25 (exports org.egothor.stemmer)
lucene/CHANGES.txt:15072-15074 (at a9779ef3c193) ('SOLR-3737: StempelPolishStemFilterFactory loaded its stemmer table incorrectly. Also, ensure immutability and use only one instance of this table in RAM (lazy loaded) since it's quite large.')
lucene/analysis/stempel/src/test/org/egothor/stemmer/TestStemmer.java:151-168 (at a9779ef3c193) (assertTrieContents builds {trie, reduce(Optimizer), reduce(Optimizer2), reduce(Gener), reduce(Lift(true)), reduce(Lift(false))} from one trie, then checks getFully and getLastOnPath of every one, the original included); lucene/analysis/stempel/src/java/org/egothor/stemmer/Compile.java:145-169 (trie = trie.reduce(..) for every method letter: the input is discarded)
The finding in 30 seconds: lucene/analysis/stempel/src/java/org/egothor/stemmer/Lift.java:82-103 (at a9779ef3c193) (javadoc, optimize, orows = orig.rows at :92), :114-146 (liftUp writes c.skip, c.cnt, c.cmd, c.ref)
lucene/analysis/stempel/src/java/org/egothor/stemmer/Gener.java:71-96 (at a9779ef3c193) (javadoc '@return a new optimized Trie' at :75; orows = orig.rows at :81), :105-129 (eat: :112, :116, :120-122)
lucene/analysis/stempel/src/java/org/egothor/stemmer/Reduce.java:88-128 (at a9779ef3c193) (removeGaps; Remap copies every cell with new Cell); lucene/analysis/stempel/src/java/org/egothor/stemmer/Optimizer.java:86; lucene/analysis/stempel/src/java/org/egothor/stemmer/Trie.java:361-369; lucene/analysis/stempel/src/java/org/egothor/stemmer/MultiTrie2.java:276-284
lucene/analysis/stempel/src/java/org/apache/lucene/analysis/pl/PolishAnalyzer.java:60-63, (at a9779ef3c193) :69-96, :121, :140; lucene/analysis/stempel/src/java/org/apache/lucene/analysis/stempel/StempelPolishStemFilterFactory.java:50; lucene/analysis/stempel/src/java/org/apache/lucene/analysis/stempel/StempelStemmer.java:58-71, :79-80; lucene/analysis/stempel/src/java/org/egothor/stemmer/Diff.java:103-148 (patch semantics)
Who is affected: lucene/analysis/stempel/src/java/org/egothor/stemmer/Compile.java:145-170 (at a9779ef3c193) (trie = trie.reduce(..)); lucene/analysis/stempel/src/java/org/apache/lucene/analysis/stempel/StempelStemmer.java:79-80; lucene/analysis/stempel/src/java/org/apache/lucene/analysis/pl/PolishAnalyzer.java:69-96
Why it looks unintended: lucene/analysis/stempel/src/java/org/egothor/stemmer/Trie.java:361-369; (at a9779ef3c193) lucene/analysis/stempel/src/java/org/egothor/stemmer/Gener.java:71-76; lucene/analysis/stempel/src/java/org/egothor/stemmer/Lift.java:82-87, :105-113 (liftUp javadoc); lucene/analysis/stempel/src/java/org/egothor/stemmer/Gener.java:97-104, :112, :121-122 (eat javadoc and writes); lucene/analysis/stempel/src/java/org/egothor/stemmer/MultiTrie2.java:195; lucene/analysis/stempel/src/java/org/apache/lucene/analysis/pl/PolishAnalyzer.java:60; lucene/CHANGES.txt:15072-15074; lucene/analysis/stempel/src/java/module-info.java:25
lucene/analysis/stempel/src/test/org/egothor/stemmer/TestStemmer.java:151-168 (at a9779ef3c193)
History: a Lucene clone: git log a9779ef3c193..origin/main -- lucene/analysis/stempel (empty; origin/main c5ab3e57b9b, 2026-10-28)

Other statements trace to the hunt's working files (models, logs, reviews and briefs), which are not published.

Repro files

TestTrieReduceKeepsInput.java (JUnit test): repro/LFV-696/src/org/apache/lucene/analysis/stempel/TestTrieReduceKeepsInput.java (32 lines)

About the diagrams

Made in parallel with this doc, from the same example (the default table, 'książkami rzekami pracowali', and the one-key trie {aa -> Da}); the values come from the executed repro on main a9779ef3c193.

assets/LFV-696/bug.png (572 KB)

Built 2026-11-06 from the hunt's record of this finding and its doc file. Lucene main under test in this hunt: ea57360912ed. Every listed finding was re-run on Lucene main e1faeba08bde on 2026-11-02.