Stempel tables compiled with Gener (`G`) can fail to load or lose stems: `Row.store` counts the empty cells it does not write

Stempel's Compile tool turns a dictionary into a stemmer table, and StempelStemmer.load reads it back. With the G reduction (Gener) in the method string, the table can fail to load with java.io.EOFException, or load without error and leave some words unstemmed. The cause is a writer/reader mismatch: Row.store writes the number of all cells of a row and then skips the empty cells that Gener can leave behind, while Row(DataInput) reads exactly the declared number of records and so runs into the next row. Two 2-line dictionaries show both outcomes; the bundled Polish table (-0ME2, no G) is not affected. Present since the module's first import; the fix is one line.

• Stemmer table: a trie stored as a list of rows. A row maps a character to a cell; a cell holds a patch command (how to turn the word into its stem, for example Da: delete one letter) and/or a reference to the next row. StempelStemmer.stem walks the trie and applies the last command on the path.
• Compile: org.egothor.stemmer.Compile "is used to compile a stemmer table" (Compile.java:67). Its first argument is a method string; the letters G, L, E, 2 and 1 apply reductions in order (:145-172), then the trie is written with Trie.store (:174-179).
• Gener (G): "Return a Trie with infrequent values occurring in the given Trie removed" (Gener.java:71-76). In each row it drops every command whose use count is below frame = sum / 10, the row's total use count divided by 10 and rounded down (:106-109, :116; c.cmd = -1, :120-123), drops rows left with nothing, and clears references to them (c.ref = -1, :110-114).
• Empty cell: a cell with no command and no reference (both -1). Gener leaves them; public Row.setCmd(way, -1) makes one too (Row.java:108-118), and so does Row.setRef(way, -1) on a new cell or a cell with only a reference (:126-135).

1. Row.store writes os.writeInt(cells.size()), the number of all cells, including empty ones (Row.java:232), then skips every empty cell (if (e.cmd < 0 && e.ref < 0) continue;, :237-239).
2. Row(DataInput) reads exactly the declared number of 18-byte records (a char and four ints, :78-88). The format has no byte length or end marker per row, only that count: Trie(DataInput) reads the rows one after another from the same stream (Trie.java:84-93).
3. So for every skipped cell the reader takes the next 18 bytes of the stream as one more record. If the stream ends first, load throws EOFException; otherwise the row gets a junk cell, the next row's count is read from the wrong offset, and later rows can be misread too. Lookups through those rows return wrong or no commands.
4. Row.getCells(), "Return the number of cells in use" (:137-153), counts exactly the cells store writes, and Compile prints it as cells after every reduction (Compile.java:171, Trie.java:372-385).

Example A, x xa xb … xt and a ab, method G: Gener empties root cell x; the 64-byte file declares the root row as 2 cells but holds 1 record (a). On load the root row gets a junk second cell (key \u0000, command 65,634) made from the next row's count and record, and the next row reads its count as 0: stem("ab") is null instead of a, with no error. Example B, a ab and ac aca … act: the short row (b plus an empty c) is the last one, and load throws EOFException at Row.java:80. For both, Compile prints cells 2 while the file declares 3. Without a reduction letter both load (446 bytes) and stem ab to a. The bundled table (stemmer_20000.tbl) has 0 empty cells among 121,822 and loads; reduced with Gener it has 38,322 and throws EOFException.

• Who is exposed. Users who build their own stempel table with Compile and a method string containing G (for example G, -EG, -0ME2G), or who reduce a trie with Gener, or clear a command or reference with Row.setCmd(way, -1) or Row.setRef(way, -1), and store it through the exported package org.egothor.stemmer (module-info.java:25). Custom tables are an intended use: StempelStemmer says "there is nothing language-specific here" (StempelStemmer.java:32-33).
• Opt-in. The bundled table stemmer_20000.tbl was compiled with -0ME2 (its header) and has no empty cells, so PolishAnalyzer (PolishAnalyzer.java:49, :88-89) and StempelPolishStemFilterFactory (same table, :50) are not affected.
• Not size-dependent. Small and large tables fail when Gener leaves an empty cell: two-line dictionaries, 11 Polish lemmas compiled with -0ME2G (the bundled method plus G) and the bundled table reduced with Gener (38,322 empty cells). Not every G table is hit: -0MG and -0MLG on the same lemmas leave no empty cell and load. In the finder's random sweep of 3,000 small tries, Gener left empty cells after 1G, 2G, EG and LG in 4,025 (trie, sequence) cases (one trie can count under several sequences), and every one failed the store/load round trip (3,294 EOFException, 731 silent; the finder's figure only).
• What the user sees. Either an EOFException when the application loads its table (most cases of the finder's synthetic sweep), with no hint that G is the cause (Compile reports success). Or no error at all, and some words come back unstemmed or wrongly stemmed, so matching on stems silently changes.
• Run-to-run variation (9.11.0 and later). Since #13420 (in 9.11.0 and 10.0.0) Row.store writes cells in the key order of an HPPC hash map whose iteration seed depends on how many maps the JVM created before, so whether a given table throws or loads silently can change between runs (executed with an independent reproducer's dictionary; the cause is read from the code). The repro's examples are built not to depend on it.
• Old files stay broken. The format has no byte length or end marker per row, so it gives a reader no general, deterministic way to find where a short row ends; tables written with empty cells must be recompiled after the fix.
• Likelihood: rare. Lucene does not document G, the bundled table does not use it, and no prior report was found in the searches listed below (the code is 16 years old).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: command of 'x' in the second stored row expected:<3> but was:<-1>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:647)
       org.egothor.stemmer.TestRowStoreEmptyCell.testRowWithEmptyCellRoundTrip(TestRowStoreEmptyCell.java:43)

• One format, two halves. Row.store is "Write the contents of this Row to the given output stream" (Row.java:225-230) and Row(DataInput) is "Construct a Row object from input carried in via the given input stream" (:72-77). A count that declares more records than are written produces data that the only reader cannot parse.
• The right count exists. getCells() counts exactly the written cells (:137-153) and Compile prints it as cells after every reduction (Compile.java:171, Trie.java:372-385). The skip at :237-239 is deliberate; the count at :232 is the part that disagrees.
• The trigger is the tool's purpose. Compile "is used to compile a stemmer table" (:67), handles G (:148-151) and stores the result (:174-179); StempelStemmer.load is "Load a stemmer table from an inputstream" (StempelStemmer.java:57). Gener, Trie.reduce, Trie.store, Row and Row.setCmd are public, in an exported package with no @lucene.internal or @lucene.experimental mark.
• Gener is meant to keep a usable table. Its javadoc says it removes infrequent values (Gener.java:71-76), and TestStemmer asserts that a Gener-reduced trie answers its keys correctly in memory (TestStemmer.java:157); only the stored form breaks.
• No test or discussion says otherwise. No upstream test stores a Gener-reduced trie (TestCompile uses test, -test and Mtest, TestCompile.java:79, 93, 107: Compile reads the leading - and M, but none of them holds a reduction letter G, L, E, 2 or 1, Compile.java:88-100, :147-169); the count and the skip are unchanged since 2010 with no issue or comment about them.

The strongest case (built by the intended-behaviour checker and the cross-model reviews): org.egothor.stemmer is forked code, "taken virtually unchanged from the Egothor project" (overview.html:40-41), and on #8841 a maintainer (@jpountz) wrote "I think we should avoid applying these kinds of modifications to forked code". Lucene does not document G: Compile's javadoc points to a list "in the package description" (Compile.java:77) that does not exist (package-info.java is one line). The bundled table does not use G, no test stores a Gener-reduced trie, the Lucene.NET port and the original Egothor2 code have the same lines, and no prior report was found in 16 years of history.

Why it fails: all of that is about priority, not intent. A writer that declares N records and writes fewer has no intended reading, and the module's own getCells() and printInfo say which count is right. The #8841 comment was about widening an API (Trie.add visibility), not a correctness fix, and the module is maintained (#13420 rewrote Row's map in 2024 and kept the count and the skip). Verdict: not intended, a defect (the cross-model reviews rated the intended argument weak). Not security: only whoever builds the table can trigger it.

Fix A was validated in a scratch worktree at a9779ef3c193 with the whole :lucene:analysis:stempel:test task, assertions on and off, plus the three repro classes; fix B the same with assertions on. The other options were not run.

Row.store writes getCells(), which counts the cells with a command or a reference: exactly the cells the loop writes (+1/-1 in Row.java).

   public void store(DataOutput os) throws IOException {
-    os.writeInt(cells.size());
+    os.writeInt(getCells()); // the number of cells written below
     Iterator<CharCursor> i = cells.keys().iterator();

Trade-offs

• A row without empty cells gets the same count and the same records as today, so the bundled table and every table compiled without G store the same content and load the same trie, and every existing reader reads the new files. The bytes are not guaranteed identical: since 9.11.0 each key iteration advances the row map's iteration seed (CharObjectHashMap.java:432-434, :530-531), so stock Row.store already varies the record order from one store to the next, and fix A's extra pass moves it again (executed: the bundled table stored twice, stock and with fix A, gives different digests but the same size and the same content).
• Empty cells are absent after loading, so a loaded row is not identical to the stored one: getCnt returns -1 for them instead of their count (Row.java:209-212), and print or a later reduction can see that. Lookups answer the same (getCmd and getRef return -1 for a missing cell, Row.java:197-200, :220-223).
• One extra pass over a row's cells per stored row; store runs only when a table is written, never during analysis. Not benchmarked.
• Tables already written with empty cells must be recompiled: the format has no byte length or end marker per row, so a reader has no general, deterministic way to repair them. Worth a sentence in CHANGES.
• The fix makes a loaded table answer like the Gener-reduced trie that was stored; it does not undo what Gener itself removes (on the bundled table, Gener in memory changes 193,552 of 366,667 probe answers).

What the checks validated

• Scratch worktree at a9779ef3c193, gradle, seed DEADBEEF: the paste-ready test fails 3 of 3 on stock main with assertions on and off; with fix A the whole :lucene:analysis:stempel:test task passes, 8 classes, 29 tests, 0 failures, with assertions on and off: the module's 19 upstream tests plus the 10 repro tests (the paste-ready test 3, the finder's 4, the reproducer's 3).
• With the paste-ready test, ecjLintMain, ecjLintTest and checkGoogleJavaFormat pass.
• The repro runner: the paste-ready test 3 of 3 pass (assertions on and off); the finder's round-trip sweep finds 0 failures with it.

Remove if (e.cmd < 0 && e.ref < 0) continue; (Row.java:237-239, -4 lines), so cells.size() is right again.

Trade-offs

Empty cells are stored too, 18 bytes each (about 690 KB more for the bundled table reduced with Gener, 38,322 empty cells); the loaded trie then equals the stored one cell for cell, where fix A drops the empty cells. Both repair the format and give the same lookups; equally format-compatible; it drops the author's deliberate skip.

What the checks validated

The same three repro classes and the whole :lucene:analysis:stempel:test task with assertions on: 8 classes, 29 tests, 0 failures; not run with assertions off.

Remove a cell when Gener clears its last command or reference.

Trade-offs

Misses the public Row.setCmd(way, -1) path, and Gener.java is also the fix site of a sibling stempel bug (Lift and Gener rewriting their input trie). Not validated.

Make the reader tolerate short rows.

Trade-offs

The format has no byte length or end marker per row, only the wrong count, so it gives the reader no general, deterministic way to tell where a short row ends. Not attempted.

• Round-trip check. For each trie and each reduction sequence the harness kept the reduced trie in memory, stored it, loaded it back and compared the answer for every key. Sequences without Gener all round-tripped. A random sweep (3,000 tries over 26 letters, keys of length up to 3, up to 80 keys, reduction sequences of length up to 2) found every Gener sequence that left an empty cell failing: 4,025 (trie, sequence) cases after 1G, 2G, EG and LG, 3,294 with EOFException and 731 loading with changed answers. With the one-line fix shadowed in, both counts were 0.
• Wider inputs. The bundled table: 0 empty cells, round-trips; after Gener 38,322 empty cells and EOFException. Compile on 11 Polish lemmas: -0ME2G, -0M2G and -0MEG failed, -0MG and -0MLG left no empty cell on that table and loaded.
• Reduction. The hypothesis came while modelling the store/load step from the code and was witnessed by the harness, then reduced to a public-API JUnit on the stock main jar (2026-10-27). This doc's paste-ready test uses two smaller examples.