The default codec's file-extension table is stale: seven extensions it writes are missing, and the vector row lists .veq instead of .vemf

The default codec's package documentation (codecs/lucene104/package-info.java, the File Formats overview) says it "defines the index file formats used in this version of Lucene" (:50), and its "Summary of File Extensions" table "summarizes the names and extensions of the files in Lucene" (:211, table :213-319). The table lists the metadata files of norms, doc values, points and vectors (.nvm, .dvm, .kdm, .vem) but not seven extensions the default codec writes for ordinary documents: .fdm, .tvm, .tmd, .psm, .dvs, .dvp and .vemf.

Its vector row (:312-318) links the default Lucene99HnswVectorsFormat and reads .vec, .vem, .veq, vex: that format writes .vec, .vemf, .vem and .vex, .veq comes only from the optional scalar-quantized formats, and vex has no dot. The default postings format's "Files and detailed format" list (Lucene104PostingsFormat.java:81-89) names .tim, .tip, .doc, .pos and .pay but not the .tmd and .psm files the same format writes.

Lucene writes, reads, merges and checks all these files correctly; nothing reads the table at runtime. The table matched the codec through 8.4.1. None of the six format changes since 8.5.0 (2020) that added the missing files put them in the table: five touched no package-info.java, and #13585 copied the whole table into its new lucene912 package without a .psm row. The vector row's current text was written by #12582 (9.9.0), when the HNSW format could still write .veq; from 9.0.0 through 9.8.0 the row read .vec, .vem and lacked .vex. Documentation only; the fix is javadoc text.

A stored StringField with term vectors, a NumericDocValuesField and a 2-dimensional KnnFloatVectorField give 22 files: segments_1 and 21 segment files.

With compound files the directory holds only _0.cfe, _0.cfs, _0.si and segments_1, all listed.

It links Lucene99HnswVectorsFormat, the default (Lucene104Codec.java:123), with the cell .vec, .vem, .veq, vex and the text ".vem the vector metadata, .veq the quantized vector data, and .vex the hnsw graph data". In lucene/core, .veq (with .vemq) is defined only by the optional Lucene104ScalarQuantizedVectorsFormat (:107-108); the read-only backward-codecs Lucene99ScalarQuantizedVectorsFormat (:53-54) defines the same two for old indexes. The default format writes .vec and .vemf through its flat format and .vem and .vex itself (:89-90).

"Files and detailed format:" .tim, .tip, .doc, .pos, .pay. The format also writes .psm and, through its terms dictionary (:410-413), .tmd; .tim and .tip, also terms-dictionary files, are listed.

• Who is hit. Anyone who looks an index file up in the File Formats overview, or builds on it: sizing an index by file type, a backup, copy or inspection tool that selects files by extension, a port of the format. Lucene users who only call its APIs are unaffected, and Lucene's own file lists (SegmentInfo.files(), the commit's file names) are correct.
• Default vs opt-in. The default codec and ordinary documents: .fdm, .tmd and .psm are written for nearly every segment (stored fields and postings), .dvs and .dvp for every segment with doc values, .vemf for every segment with vectors, .tvm with term vectors.
• When the files are visible. Flushed segments are compound by default (IndexWriterConfig.java:99, DocumentsWriterPerThread.java:643), but a merge writes any segment above 64 MB as separate files (CompoundFormat.java:42, :163-179; the default TieredMergePolicy measures bytes, MergePolicy.java:598-600). So any default index with a merged segment above 64 MB shows them in its directory.
• Scale. None needed: one document shows 7 of 21 extensions unlisted.
• What it does not do. No wrong results, no index change, no runtime path. Every missing file except .psm is documented on its own format page, though not every such page is reachable from the table: .tmd's page (Lucene103BlockTreeTermsWriter) is one link away from the table's postings link, and .vemf's page (Lucene99FlatVectorsFormat) cannot be reached from the vector row's link, whose page links only the abstract FlatVectorsFormat (Lucene99HnswVectorsFormat.java:41). The searches found no report of it although it dates from 8.5.0 (2020), which suggests the audience is small. Impact class minor, priority low.

• The test embeds the table, so it checks today's text, not a parse of it. That its 25 extensions equal the table's was checked mechanically at 645d4fe14a20, 10.5.2, 10.6.0 and later main (a control set compares different, so the check is not vacuous).
• It asserts the missing extensions only. The vector row and the postings list rest on the source lines (see the explanation above) and on the version-aware check, which parses the table and the list from source, indexes 300 documents non-compound and compound, and fails four ways on main (shown in the full output).
• Even simpler for a reader: put the table (package-info.java:213-319) next to a directory listing of a non-compound segment.
• Results: fails 1 of 1 on main with assertions on and off (seed DEADBEEF; the test data has no random values, though LuceneTestCase still randomizes things such as locale and time zone), standalone and through gradle; on the 10.4.0, 10.5.2 and 10.6.0 release jars it reports 5, 6 and 7 missing extensions.
• As a regression test: the same test with the seven extensions added to its set passes on the fixed tree and fails the next time the default codec writes an extension the set lacks. It keeps veq, which fix A moves out of the vector row's extension cell, so its set has 32 extensions and the fixed table 31 (dropping veq would make them equal; not run). Like the repro it checks only that every written extension is in its set, not the vector row, the dots or the postings list.

cp TestLucene104ExtensionTable.java $LUCENE/lucene/core/src/test/org/apache/lucene/codecs/lucene104/
cd $LUCENE && ./gradlew :lucene:core:test --tests org.apache.lucene.codecs.lucene104.TestLucene104ExtensionTable -Ptests.seed=DEADBEEF
# assertions off: add -Ptests.asserts=false (same result)

TestLucene104ExtensionTable > testTableListsEveryExtensionTheDefaultCodecWrites FAILED
    java.lang.AssertionError: written extensions missing from the extensions table expected:<[]> but was:<[dvp, dvs, fdm, psm, tmd, tvm, vemf]>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:33A5926AE3A763FA]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:120)
        at org.apache.lucene.codecs.lucene104.TestLucene104ExtensionTable.testTableListsEveryExtensionTheDefaultCodecWrites(TestLucene104ExtensionTable.java:64)

# release jars: 10.4.0 [fdm, psm, tmd, tvm, vemf]; 10.5.2 [dvs, fdm, psm, tmd, tvm, vemf]; 10.6.0 [dvp, dvs, fdm, psm, tmd, tvm, vemf]

# version-aware check (parses the table and the postings list from source), main:
FAIL A: written but absent from the table (excluding points files): [dvp, dvs, fdm, psm, tmd, tvm, vemf]
FAIL B: vector row lists [.vec, .vem, .veq, vex] but the default vectors format wrote [vec, vem, vemf, vex]
FAIL C: table extension cells without a leading dot: [vex]
FAIL D: written by Lucene104PostingsFormat but absent from its 'Files and detailed format' list [doc, pay, pos, tim, tip]: [psm, tmd]
failures=4

• Contract text. package-info.java:50: "This document defines the index file formats used in this version of Lucene"; :211: "The following table summarizes the names and extensions of the files in Lucene". The table was added (LUCENE-1841, #2916, 2009) so readers can look files up by extension.
• No rule explains the omissions. The table lists the metadata files of norms, doc values, points and vectors (.nvm, .dvm, .kdm, .vem), but not those of stored fields, term vectors, terms, postings and flat vectors.
• History shows omissions, not decisions. None of the six changes that added the missing files put them in the table: five touched no package-info.java, and #13585, which copied the whole table into its new lucene912 package, gave .psm no row and documented it only in the constant's javadoc. The PR discussions the follow-up read do not mention the table. A history search (git log -G over every package-info.java) found no version of the table that ever had a cell with any of the seven extensions. vex was dotted in 10.2.x and came back undotted from an older copy in 10.3.0.
• Precedent. #13801 (romseygeek, merged 2024-09-18, "Correct Point file extensions in Codec javadocs") replaced the stale points cell .dii, .dim with .kdd, .kdi, .kdm. Other file-format doc fixes: LUCENE-7532 (#8583), #15625.
• Tests. No test or build check was found that compares the overview with the files written (searched lucene/*/src/test, build-tools and gradle at 645d4fe14a20).

Strongest case for "intended" (the refute reviewer, the intended-lens cross-model reviewer and the devil's-advocate verifier; all rated it moderate): the page "attempts to provide a high-level definition" (:54), the table only "summarizes", the details are "provided in their linked pages" (:188), and those pages document the missing files (all but .psm); .veq is a real extension of this version (the optional scalar-quantized formats); and the codec is @lucene.experimental (Lucene104Codec.java:53).

Why it fails: a summary may be short, but not wrong: the vector row states something false about the format it links, vex is a typo, and listing some formats' metadata files but not others is inconsistency, not brevity. @lucene.experimental allows the format to change, not the current one to be described wrongly. The linked pages do not close the gap either: .vemf's page cannot be reached from the vector row's link, .tmd's is one link further on, and .psm has no page. Upstream treated exactly this as a defect in #13801. Every verifier and the adjudication rejected the "intended" reading.

Three ways to resolve it; A was validated. Targets: main and branch_11x, plus branch_10_6 if wanted (branch_10x was decommissioned, #16911). In 10.6.0 package-info.java is the same file and the postings list the same text, so the diff should apply there too (not tried).

Optional extras: the frozen backward-codecs lucene99 and lucene103 package docs carry the same vector row; fixing them is optional, as they describe old codecs.

Four new rows (.fdm Field Metadata, .tmd Term Metadata, .psm Postings Metadata, .tvm Term Vector Metadata), each linked to its format like its neighbours; the doc-values row becomes .dvd, .dvm, .dvs, .dvp; the vector row becomes .vec, .vemf, .vem, .vex, describes .vemf as the raw vector metadata and .vem as the HNSW graph metadata, and says the optional scalar-quantized formats add .veq and .vemq. In Lucene104PostingsFormat, the "Files and detailed format" list gains .tmd (with a link to Lucene103BlockTreeTermsWriter) and .psm (maximum impact sizes and the end pointers of the .doc, .pos and .pay files, as Lucene104PostingsWriter.java:711-720 writes). Two files, +28/-4 lines.

diff --git a/lucene/core/src/java/org/apache/lucene/codecs/lucene104/Lucene104PostingsFormat.java b/lucene/core/src/java/org/apache/lucene/codecs/lucene104/Lucene104PostingsFormat.java
index f2fb84f3a7e..df59b54aa43 100644
--- a/lucene/core/src/java/org/apache/lucene/codecs/lucene104/Lucene104PostingsFormat.java
+++ b/lucene/core/src/java/org/apache/lucene/codecs/lucene104/Lucene104PostingsFormat.java
@@ -83,9 +83,12 @@ import org.apache.lucene.util.packed.PackedInts;
  * <ul>
  *   <li><code>.tim</code>: <a href="#Termdictionary">Term Dictionary</a>
  *   <li><code>.tip</code>: <a href="#Termindex">Term Index</a>
+ *   <li><code>.tmd</code>: Term Metadata, see {@link Lucene103BlockTreeTermsWriter}
  *   <li><code>.doc</code>: <a href="#Frequencies">Frequencies and Skip Data</a>
  *   <li><code>.pos</code>: <a href="#Positions">Positions</a>
  *   <li><code>.pay</code>: <a href="#Payloads">Payloads and Offsets</a>
+ *   <li><code>.psm</code>: Postings metadata (maximum impact sizes and the end pointers of the
+ *       .doc, .pos and .pay files)
  * </ul>
  *
  * <a id="Termdictionary"></a>
diff --git a/lucene/core/src/java/org/apache/lucene/codecs/lucene104/package-info.java b/lucene/core/src/java/org/apache/lucene/codecs/lucene104/package-info.java
index ebcf8c1e5e4..1a90a706783 100644
--- a/lucene/core/src/java/org/apache/lucene/codecs/lucene104/package-info.java
+++ b/lucene/core/src/java/org/apache/lucene/codecs/lucene104/package-info.java
@@ -255,6 +255,11 @@
  * <td>The stored fields for documents</td>
  * </tr>
  * <tr>
+ * <td>{@link org.apache.lucene.codecs.lucene90.Lucene90StoredFieldsFormat Field Metadata}</td>
+ * <td>.fdm</td>
+ * <td>Metadata about the stored fields and their index</td>
+ * </tr>
+ * <tr>
  * <td>{@link org.apache.lucene.codecs.lucene104.Lucene104PostingsFormat Term Dictionary}</td>
  * <td>.tim</td>
  * <td>The term dictionary, stores term info</td>
@@ -265,6 +270,11 @@
  * <td>The index into the Term Dictionary</td>
  * </tr>
  * <tr>
+ * <td>{@link org.apache.lucene.codecs.lucene104.Lucene104PostingsFormat Term Metadata}</td>
+ * <td>.tmd</td>
+ * <td>Per-field term statistics and the location of the Term Index</td>
+ * </tr>
+ * <tr>
  * <td>{@link org.apache.lucene.codecs.lucene104.Lucene104PostingsFormat Frequencies}</td>
  * <td>.doc</td>
  * <td>Contains the list of docs which contain each term along with frequency</td>
@@ -280,13 +290,18 @@
  * <td>Stores additional per-position metadata information such as character offsets and user payloads</td>
  * </tr>
  * <tr>
+ * <td>{@link org.apache.lucene.codecs.lucene104.Lucene104PostingsFormat Postings Metadata}</td>
+ * <td>.psm</td>
+ * <td>Small metadata about how postings are encoded</td>
+ * </tr>
+ * <tr>
  * <td>{@link org.apache.lucene.codecs.lucene90.Lucene90NormsFormat Norms}</td>
  * <td>.nvd, .nvm</td>
  * <td>Encodes length and boost factors for docs and fields</td>
  * </tr>
  * <tr>
  * <td>{@link org.apache.lucene.codecs.lucene90.Lucene90DocValuesFormat Per-Document Values}</td>
- * <td>.dvd, .dvm</td>
+ * <td>.dvd, .dvm, .dvs, .dvp</td>
  * <td>Encodes additional scoring factors or other per-document information.</td>
  * </tr>
  * <tr>
@@ -295,6 +310,11 @@
  * <td>Stores offset into the document data file</td>
  * </tr>
  * <tr>
+ * <td>{@link org.apache.lucene.codecs.lucene90.Lucene90TermVectorsFormat Term Vector Metadata}</td>
+ * <td>.tvm</td>
+ * <td>Metadata about the term vectors and their index</td>
+ * </tr>
+ * <tr>
  * <td>{@link org.apache.lucene.codecs.lucene90.Lucene90TermVectorsFormat Term Vector Data}</td>
  * <td>.tvd</td>
  * <td>Contains term vector data.</td>
@@ -311,10 +331,11 @@
  * </tr>
  * <tr>
  * <td>{@link org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsFormat Vector values}</td>
- * <td>.vec, .vem, .veq, vex</td>
+ * <td>.vec, .vemf, .vem, .vex</td>
  * <td>Holds indexed vectors; <code>.vec</code> files contain the raw vector data,
- * <code>.vem</code> the vector metadata, <code>.veq</code> the quantized vector data, and <code>.vex</code> the
- * hnsw graph data.</td>
+ * <code>.vemf</code> the raw vector metadata, <code>.vem</code> the hnsw graph metadata, and
+ * <code>.vex</code> the hnsw graph data. The optional scalar-quantized formats add
+ * <code>.veq</code> (quantized vector data) and <code>.vemq</code> (its metadata).</td>
  * </tr>
  * </table>
  *

Trade-offs

• Covers every gap; the descriptions are short and match the writers. The .psm description is the only one that is new text (no format page describes the file today).
• The table still has to be kept up to date by hand. A test like the repro, with its set updated, would catch the next gap (optional).
• For a branch_10_6 backport the same diff should apply (not tried); .dvp exists there (10.6.0).

What the checks validated

Validated in a scratch worktree of 645d4fe14a20 with the diff applied (removed afterwards), seed DEADBEEF, 2026-11-01:

• The fixed text matches the files. The version-aware check, which parses the table and the postings list straight from the patched source and indexes 300 documents with the default codec (non-compound and compound), reports 0 failures with assertions on and off; on the stock text it reports 4 (missing extensions, vector row, undotted vex, postings list).
• Suite: ./gradlew :lucene:core:test --tests org.apache.lucene.codecs.lucene104.TestLucene104PostingsFormat --tests org.apache.lucene.codecs.lucene104.TestLucene104ExtensionTableFixed: 35 tests (34 of the suite, 2 of them skipped by the framework, and the repro with the seven extensions added to its set), BUILD SUCCESSFUL with assertions on and off.
• Static checks and docs: :lucene:core:checkGoogleJavaFormat passes with the diff unchanged; :lucene:core:check -x test (ecjLint, forbiddenApis, google-java-format, the regenerate checksums) and :lucene:core:renderJavadoc pass; the rendered page shows the four new rows and the corrected doc-values and vector rows, and the rendered postings list shows .tmd and .psm.
• The unchanged repro still fails on the patched tree, as it must: it embeds the old table. Through gradle on stock 645d4fe14a20 it fails with assertions on and off.
• The full Lucene test suite was not run (javadoc-only change).

Correct the vector row (.vec, .vemf, .vem, .vex and the .veq note) and the postings list; leave the missing metadata rows.

Trade-offs

Smallest diff, but the table stays incomplete and inconsistent (some formats' metadata files listed, others not), which is the larger part of the defect. Not run.

Fix the vector row and the postings list, and change :211 to say the table lists the main files of each format and that each linked page lists its metadata files.

Trade-offs

Nothing left to drift for future metadata files, but readers looking up an unknown file by extension still have to search the format pages, which is what the table is for. Not run.

• Hunt unit, 2026-10-28: the table's extension cells and Lucene104PostingsFormat's "Files and detailed format" list were extracted mechanically from the javadoc source. The extensions written were collected from the real files (top level and inside compound files) of 91 generated indexes: 11 fixed configurations (two of them with a scalar-quantized vectors format) and 80 seeded random feature mixes (stored fields, term vectors, postings with positions, offsets and payloads, norms, all five doc-values types dense and sparse with and without a skip index, points, float, byte and float16 vectors, deletions, soft deletes, doc-values updates, compound files, a file-system directory). The two sets were compared both ways, with assertions on and off.
• Result: written but not in the table [.dvp, .dvs, .fdm, .psm, .tmd, .tvm, .vemf]; in the table but never written by the default format [.veq]; postings list misses [.psm, .tmd]. On the fixed text the same extraction, run as a separate 41-configuration sweep (the 11 fixed and 30 seeded random), finds nothing missing, extra or undotted (positive control; a sweep, not a proof).
• Starting lead: a follow-up of an earlier unit that collected the set of extensions the default codec writes. This is a differential check, not model checking or a proof.