FSTMetadata.save always writes the current version, so re-saving a pre-9.0 BYTE2 FST breaks its lookups

FST.FSTMetadata#save writes the constant VERSION_CURRENT (9) into the FST header, whatever version the FST it saves is in, while FST#save copies the FST bytes of an on-heap FST (what FST.read and the FST(FSTMetadata, DataInput) constructor create) unchanged. Lucene 9.0+ still reads FST formats 6 and 7 (written by Lucene 7.0.0-8.3.1 and 8.4.0-8.11.4; older formats are rejected when loaded), and for BYTE2 labels the decoding depends on the header version: before format 8 the two label bytes are in the old order, and readLabel swaps them back only if the header says less than 8. So a char-keyed FST written by Lucene 8.11.4, loaded and saved again on 9.0 or later, comes back with header 9 (8 on 9.0-9.8) over format-7 bytes: every label is read byte-swapped and the lookups go wrong, with no error (in the repro 9 of 9 keys before, 0 of 9 after). The copy differs from the original only in the version byte. Every release from 9.0.0 to 10.5.2 and main have it; 8.11.4 re-saves the same file correctly. The same line also labels an FST built with setVersion(FST.VERSION_90) as version 9 instead of 8.

• FST: Lucene's compact map from a sequence of labels to an output (terms index, suggesters, dictionaries). It can be written to a file with FST#save and read back with FST.readMetadata plus the FST constructor, or FST.read(Path).
• BYTE2: the input type whose labels are 2-byte values, used for char-keyed FSTs built over UTF-16 code units (Util.toUTF16). BYTE1 labels are single bytes, BYTE4 labels are vInts.
• Header version: the FST format version in the file header (CodecUtil header, a big-endian int at bytes 8-11). Lucene 7.x and 8.0-8.3 wrote 6, 8.4-8.11 wrote 7, 9.0-9.8 wrote 8, 9.9 and later write 9.

1. Lucene 8.x wrote BYTE2 labels with the big-endian writeShort. LUCENE-9047 made DataOutput little-endian in 9.0 (MIGRATE.md:1179-1183), and FSTCompiler.writeLabel writes BYTE2 labels with out.writeShort since (FSTCompiler.java:585-587).
2. readMetadata accepts formats 6 to 9 (:457-459) and keeps the file's version (:493). readLabel swaps BYTE2 labels when that version is below 8: if (metadata.version < VERSION_LITTLE_ENDIAN) v = Short.reverseBytes(in.readShort()) & 0xFFFF; (:553-554). It is the only version-dependent read.
3. FST#save writes the metadata, then the FST bytes (:524-527): for an on-heap FST, which FST.read and the FST(FSTMetadata, DataInput) constructor create (:419-420, :538-543), the loaded bytes unchanged (OnHeapFSTStore.java:84-90; an off-heap FST cannot be saved, OffHeapFSTStore.java:64-66). FSTMetadata#save writes CodecUtil.writeHeader(metaOut, FILE_FORMAT_NAME, VERSION_CURRENT) (:1214), ignoring its own version field.
4. Reloading the copy, the header says 9, readLabel no longer swaps, and every BYTE2 label is read with its bytes reversed. Util.get returns null for all 9 keys of the repro; nothing throws.

The 8.11.4 fixture (16 metadata bytes, 64 FST bytes, 9 keys): the original finds 9 of 9 keys with getVersion() = 7. After save the metadata differs in one byte (offset 11, 07 → 09) and the FST bytes are identical; the copy finds 0 of 9 keys with getVersion() = 9, get("ab") = null and get("愀戀") = 1, the byte-swapped key. Its enumeration lists 愀戀 → 1 ... 挀愀琀 → 5. With BYTE1 or BYTE4 labels the same steps keep every key. The 0 of 9 (and 0 of 45 in the history probe) holds for these fixtures, not for every key of every BYTE2 FST: a key made only of UTF-16 units whose two bytes are equal survives (an executed check: an 8.11.4 FST {a → 1, U+6161 → 2} re-saved on main gives get("a") = null and get("慡") = 2), the empty input keeps its output (stored in the metadata, its lookup reads no label: FST.java:593-596, Util.java:47-65), and byte-swapped keys can resolve instead.

FSTCompiler.Builder#setVersion (FSTCompiler.java:335-348) accepts VERSION_90 (8) to VERSION_CURRENT; the compiler honours it (no continuous-arc nodes below 9, :543) and the in-memory version is 8, but save writes 9. After readMetadata, getVersion() returns 9, and a Lucene 9.8 reader rejects the file with IndexFormatTooNewException, although its content is format 8: with only the header int patched to 8, 9.8.0 reads a 50,000-term FST with 0 wrong lookups.

• Who is exposed. Applications that persist standalone FSTs (FST.save(Path) / FST.read(Path), or save(DataOutput, DataOutput)) with BYTE2 labels, in format 6 or 7 (written by Lucene 7.0.0 to 8.11.4), and that load such a file on 9.0 or later and save it again: a copy to another place, a migration to a new store, a cache that writes the loaded FST back. The original file keeps working; the copy is silently wrong on every 9.0+ reader, and 8.x readers reject header 8 or 9.
• No in-tree path. The suggesters that store and load FSTs (AnalyzingSuggester, FSTCompletionLookup, WFSTCompletionLookup, FreeTextSuggester, NRTSuggesterBuilder) build them with BYTE1, so a re-stored old suggester is only mislabelled. The in-tree BYTE2 FSTs (Kuromoji and Nori dictionaries and user dictionaries, NormalizeCharMap, Hunspell ConvTable) are built fresh: the bundled dictionary FSTs are regenerated from source (#12911, #12924), the user dictionaries from the user's text.
• Default, but rare. No setting is needed; the trigger is the data plus the re-save. How many applications keep pre-9.0 BYTE2 FST files and re-save them is unknown; no downstream report was found. FST and FSTCompiler are @lucene.experimental.
• Scale. Every key of the test fixtures is lost: 0 of 9 in the repro, 0 of 45 in the history probe on every release from 9.0.0 to 10.5.2 and main, for files written by 7.7.3 (format 6) and 8.11.4 (format 7). Not a rule for every BYTE2 FST: keys made only of UTF-16 units with two equal bytes (such as U+6161, executed) and the empty input can survive, and byte-swapped keys can resolve instead (get("愀戀") = 1 in the repro).
• Recoverable. The copy differs from the original only in the version byte, so restoring it gives back the original file.
• Second symptom. Needs the expert setVersion(VERSION_90); the only in-tree caller is the backward-codecs test writer Lucene90BlockTreeTermsWriter. Apart from the wrong reported version (getVersion() = 9 after readMetadata, on current readers too), its only demonstrated compatibility failure is rejection by readers capped at version 8 (9.0 to 9.8), which Lucene does not promise to support.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<1> but was:<null>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:120)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.util.fst.TestResavePre90Byte2Fst.testResaveKeepsLookups(TestResavePre90Byte2Fst.java:44)

• Reading old FSTs is offered on purpose. VERSION_START = 6 is "the version that was used when releasing Lucene 7.0" (FST.java:113-114); readMetadata accepts 6 to 9 ("we don't have back-compat promise for FSTs (they are experimental), but we are sometimes able to offer it", :457-459), and readLabel has a branch written for exactly these old BYTE2 files (:553). Main decodes the 8.11.4 file correctly: 9 of 9 keys.
• The documented meaning of the version. FSTMetadata#getVersion: "Returns the version constant of the binary format this FST was written in" (:1191-1195). save + readMetadata turns 7 into 9 (and 8 into 9) while the bytes stay as they were. Kuromoji's and Nori's TestTokenInfoDictionary.testBinaryFSTIsLatestFormat (#12911) read the bundled FST's header with FST.readMetadata(...).getVersion() to check that its format is current: Lucene's own tests use the header version to tell what format the bytes are in.
• save has no precondition. "Save the FST to DataOutput" (:518-523) and "Writes an automaton to a file" (:529); the FSTCompiler javadoc lists "save it to other DataOutput, and load it later and use it" as a use (FSTCompiler.java:61-62). TestFSTs.testSaveDifferentMetaOut (:1204-1243) asserts that load, save and reload keep the lookups, for a current-format FST. No test re-saves an older-format FST or checks the header that save writes for one.
• Design intent. mikemccand on the version bump in #12748 (2023-11-06): if the version is bumped, an older reader "would throw an understandable error". Here a newer reader silently misreads, because the header claims a newer format than the bytes.
• History. The constant header dates from 2010 and was harmless until LUCENE-9047 (b84e0c272b6, 9.0.0) made BYTE2 decoding depend on the version; that commit did not touch the save line, and its PR lists "adds backwards compatibility for FST" for reading. #12908 (9.9.1) added setVersion for a test writer of "data in the 9.8 format" without threading it into the header.
• Precedent. LUCENE-8844 (#9888, 2019) treated an encoding change without a header bump, which made older readers "exhibit strange behavior", as a bug. When a break was intended, the project said so (LUCENE-3681, 2012: users "will have to rebuild"); no CHANGES, MIGRATE or javadoc entry says that re-saving a loaded older FST is unsupported. Lucene 8.11.4 itself re-saves the file with header 7 and 9 of 9 keys.

The strongest case (a cross-model review rated it "moderate"): FST is @lucene.experimental, readMetadata disclaims a back-compat promise, setVersion is "Expert", and LUCENE-3681 once told users to rebuild old FSTs. LUCENE-9011 (#10054, 2019) closed a report of a standalone FST file that no longer loaded (IndexFormatTooOldException): jpountz answered that it "has been created by Lucene 6 or earlier". rmuir wrote on java-user@ in 2013 that "mainly FST (unfortunately) has backwards compatibility because parts of the official index format use it", that is, old formats stay readable for index files, which read their FSTs in place. So old standalone FST files are best effort, and an application should rebuild an old FST rather than copy it. Apart from the wrong reported version, the VERSION_90 header only causes a failure on readers capped at 8, which Lucene does not support.

Why it fails for the main symptom: Lucene offers this compatibility deliberately, with a read branch for exactly these files, and reads them correctly; save then turns a correctly loaded FST into a file that answers wrongly (all 9 keys of the repro "not found"), by writing a header that misdescribes bytes it copies unchanged. Best effort can justify a loud IndexFormatTooOldException, which is what LUCENE-9011 got, not silent wrong answers from a public call with no documented precondition. The index-format reason explains why old formats are read at all, not why save should mislabel them. For the second symptom the case is fair.

One option was validated with gradle in a clean checkout of ea57360912ed; the others are listed with their trade-offs and were not tested.

One line in FSTMetadata.save (fix-save-own-version.diff; with the test: fix-save-own-version-with-test.diff). For an on-heap FST FST#save copies the bytes unchanged, so the header must keep the version they are in. FSTs built with default settings have version = VERSION_CURRENT (FSTCompiler.java:250), so their files do not change; a loaded old FST is re-saved byte-identical; a setVersion(VERSION_90) FST gets header 8, as 9.0-9.8 wrote.

public void save(DataOutput metaOut) throws IOException {
  CodecUtil.writeHeader(metaOut, FILE_FORMAT_NAME, version); // was VERSION_CURRENT
  ...

Trade-offs

• A re-saved old FST stays in its old format; it is not upgraded. If a later release raises VERSION_START, such files fail loudly with IndexFormatTooOldException.
• FSTMetadata has a public constructor that takes any int version, which now goes into the header as is; an out-of-range value gives a file that readMetadata rejects. A range check in the constructor or in save would catch it early (not tested).
• The backward-codecs test data in the 9.0-9.8 format now carries FST header 8 instead of 9; its tests pass either way.

What the checks validated

• Gradle, clean checkout of ea57360912ed with the patch, seed DEADBEEF, JDK 25: the repro 3 of 3 with -Ptests.asserts=true and =false (2 of 3 fail without the patch).
• :lucene:core:test --tests 'org.apache.lucene.util.fst.*' (TestFSTs, TestFSTDirectAddressing, TestUtil, TestFSTSuffixNodeCache, TestBitTableUtil, TestGrowableByteArrayDataOutput, the repro; Test2BFST and Test2BFSTOffHeap are @Ignored) plus codecs.lucene103.blocktree.*: 44 tests, 0 failures, 6 skipped (3 @Nightly, 3 @Ignore); util.fst.* with assertions off: 40 tests, 0 failures, 6 skipped (one more needs assertions).
• The whole :lucene:backward-codecs:test, which holds the only in-tree setVersion(VERSION_90) caller and every backward index test: 57 classes, 2,339 tests, 0 failures, 184 skipped.
• :lucene:core:check -x test passes with the patch and the test (google-java-format, ecj lint, forbidden APIs).
• The byte probe: the re-saved copy is byte-identical to the original and finds 9 of 9 keys; a cross-model review's own tests pass with the same patch.

When version < VERSION_LITTLE_ENDIAN and the input type is BYTE2, write a current-format FST instead of copying, for example by rebuilding it from its own enumeration (IntsRefFSTEnum into a new FSTCompiler). Not tested.

Trade-offs

save stops being a byte copy and costs time and memory proportional to the FST. Swapping the labels in place is not practical, because labels sit at variable offsets inside node encodings. Only worth it if save should migrate old FSTs.

Throw from save when version < VERSION_LITTLE_ENDIAN and the input type is BYTE2, asking the caller to rebuild the FST. Not tested.

Trade-offs

Loud instead of silent, but it breaks copies that work with the recommended fix and leaves the second symptom. A broader "only VERSION_CURRENT" rule would break the backward-codecs writer that saves VERSION_90 FSTs.

Say on FST#save that it always writes the current version and that FSTs loaded from older formats must be rebuilt, not re-saved.

Trade-offs

Leaves the silent failure in place; listed for completeness.

Method: code reading (the study's non-formal tier). In the 2026-10-22 study, run 3 of the FST experiment, an AI tool was writing Alloy facts about the FST writer's versions and, reading the code for them, noticed that FSTMetadata.save ignores FSTCompiler.Builder#setVersion (the second symptom). The Alloy model did not produce a counterexample for it.

• A separate AI verifier (FST verifier) wrote its own tests, reproduced the mislabel on main 7cecf79523, judged it low impact, and found the main symptom, which the finder had not reported: it built a BYTE2 FST with the real 8.11.4 jar, loaded it on main (9 of 9 keys) and found 0 of 9 after a re-save. It also checked BYTE1 and BYTE4 (10 of 10) and the 9.8.0 reader.
• The study's public-list rerun reproduced it on main b83101cdd2. In this hunt it was re-verified on ea57360912ed and release jars, swept over 133 releases with a commit-level A/B, searched for prior reports, and reviewed by an independent model from another family (OpenAI GPT-5.6).