The .cfe grammar in the Lucene90CompoundFormat javadoc leaves out the footer and misstates what FileName holds

A compound segment is two files: .cfs holds the bytes of all the segment's files and .cfe is its table of contents. The class javadoc of Lucene90CompoundFormat, linked from the default codec's file-format overview as "Compound File", gives Compound (.cfs) --> Header, FileData^FileCount, Footer (Lucene90CompoundFormat.java:49 on main 645d4fe14a20) and Compound Entry Table (.cfe) --> Header, FileCount, <FileName, DataOffset, DataLength>^FileCount (:50-51), with no Footer on the .cfe line, and says each entry holds "a String with that file's name" (:65-66). The writer appends a 16-byte codec footer to .cfe (:107) and stores each name with the segment name stripped (:153), for example .fnm for _0.fnm. A decoder that follows the text stops 16 bytes before the end of every .cfe file, and a lookup by the names it read, taken as file names, fails (the segment name is missing). Lucene's reader checks the footer (Lucene90CompoundReader.java:133) and adds the segment name back (:222), so only the javadoc is wrong: the footer part in every release from 5.0.0 to 10.6.0 and on main, the FileName part from 4.0.0 on. The unused Checksum on the type line (:54) belongs to the linked CodecFooter grammar; removing it is cleanup.

Compound (.cfs) --> Header, FileData^FileCount, Footer
Compound Entry Table (.cfe) --> Header, FileCount, <FileName, DataOffset, DataLength>^FileCount

  Header --> IndexHeader                     (CodecUtil#writeIndexHeader)
  FileCount --> VInt
  DataOffset,DataLength,Checksum --> UInt64
  FileName --> String
  FileData --> raw file data
  Footer --> CodecFooter                     (CodecUtil#writeFooter)

Notes: ... Each directory entry contains a long pointer to the start of this file's data section,
the files length, and a String with that file's name. ...

The .cfe production has no Footer, although Footer is defined (:57) and used on the .cfs line (:49). Checksum (:54) is used by neither production: it is the last element of the linked CodecFooter grammar (CodecUtil.java:396).

• CodecUtil.writeIndexHeader(entries, ENTRY_CODEC, ...) (:102), entries.writeVInt(numFiles) (:117).
• Per file, in ascending size order: entries.writeString(IndexFileNames.stripSegmentName(file)), writeLong(startOffset), writeLong(length) (:153-155). stripSegmentName returns "the filename with the segment name removed" (IndexFileNames.java:133-144); it cuts at the second _ or else the first . (:119-127), so _0.fnm is stored as .fnm, while a per-field file keeps the _ of its suffix (the history probe saw _Lucene50_0.doc stored on 5.0.0-7.7.3).
• Then CodecUtil.writeFooter(entries) (:107), right after CodecUtil.writeFooter(data) for the .cfs (:106).
• The reader mirrors it: readMapping reads the string as an id (Lucene90CompoundReader.java:144), CodecUtil.checkFooter verifies the footer (:133), lookups strip the segment name from the requested name (:171, :235), and listAll adds it back (:222).

bytes 0-48     IndexHeader (49 bytes: magic, "Lucene90CompoundEntries", version, segment id, empty suffix)
byte  49       FileCount = 4 (VInt)
bytes 50-70    FileName ".fnm"  DataOffset 64   DataLength 61    (21 bytes: 1 + 4 + 8 + 8)
bytes 71-91    FileName ".fdx"  DataOffset 128  DataLength 64
bytes 92-112   FileName ".fdt"  DataOffset 192  DataLength 78
bytes 113-133  FileName ".fdm"  DataOffset 320  DataLength 157
               <- the documented grammar ends here, at byte 134 of 150
bytes 134-149  Footer: c0 28 93 e8 | 00 00 00 00 | 8-byte CRC32 checksum   (not in the .cfe grammar)

The segment id and the checksum differ per run; this one-document index's layout does not. CodecUtil.checkFooter accepts the 16 left-over bytes. The compound directory lists the sub-files as _0.fnm, _0.fdx, _0.fdt, _0.fdm. In the hunt's sweep the same holds for all 87 .cfe files of 91 generated indexes (16 bytes left over in each) and all 1720 entries (stripped names).

• Footer. LUCENE-2446 added writeFooter to both compound files in one change. On trunk (c189d0fb74d, 2014-04-01) the javadoc got , Footer on the .cfs line only, plus Checksum on the type line; the branch_4x backport (819eda82e33, 2014-04-02) put , Footer on the .cfe line instead, so 4.8.0-4.10.4 documented the .cfe footer. The LUCENE-5944 bulk backport (55594bad9b0, 2014-09-18) replaced the branch_5x text with the trunk text, so from 5.0.0 the .cfe line has no Footer. LUCENE-5969 and LUCENE-9705 copied the text into Lucene50CompoundFormat and Lucene90CompoundFormat.
• FileName. Up to 3.0.x the writer stored the full name and "with that file's name" was right. LUCENE-2455 (d6dd2e234e1, 2010, 3.1) started storing the stripped name, so that entries stay valid when a segment is renamed, and the sentence was not updated; 25a59d8902e (2012-06-11) moved it into the public format docs (4.0.0).

• Who is hit. Anyone who decodes a .cfe file without Lucene, or checks one by hand, following the class javadoc. The default codec's overview (lucene104/package-info.java) says it "defines the index file formats used in this version of Lucene" (:50), links this class as "Compound File" (:237) and says "Details on each of these are provided in their linked pages" (:188). Two outside .cfe parsers exist (a 2021 draft format description and a 2018 script); both worked around the text without reporting it.
• Default vs opt-in. The default codec; compound files are the default for new segments (IndexWriterConfig.java:96-99, DEFAULT_USE_COMPOUND_FILE_SYSTEM = true) and for small segments (package-info.java:195). Every .cfe file Lucene90CompoundFormat writes shows both parts; no setting changes its layout (a segment that is not compound has no .cfe at all).
• Scale. No runtime effect. The symptom is confined to code written from the text: a decoder stops 16 bytes early and looks for .fnm where the compound directory has _0.fnm; a tool that wrote .cfe files by the text would produce files Lucene's reader rejects at checkFooter (read, not executed). A careful reader who cross-checks the overview's footer note and the reader code can work it out, which keeps the impact minor.
• Rating. Impact class minor, likelihood plausible, priority low.

• The test sits in the codec package, like Lucene's own codec tests; Lucene90CompoundEntries is the class's ENTRY_CODEC (:79). Both methods skip the IndexHeader, read FileCount and the entries exactly as the grammar says, then compare with the real file: the first checks that nothing is left, the second compares the names with the compound directory's listAll().
• Two methods, because the javadoc makes two independent wrong statements; both read the same one-document segment.
• Results: fails 2 of 2 on main with assertions on and off, standalone (2.59 s) and through gradle, and on the published 10.6.0 jars; 5 random-seed iterations fail 10 of 10 with the same messages. The hunt's and the verifiers' larger tests also fail on 10.5.2, 9.12.3 and the branch_10_6 head. On 9.x the test needs the 3-argument getCompoundReader(dir, si, IOContext.DEFAULT).
• As a regression test: this test cannot guard the javadoc, because it transcribes the grammar. A form that pins down the corrected layout is this same test with its two assertions following the corrected grammar (CodecUtil.checkFooter after the last entry; names compared with stripSegmentName of listAll()), the fix check below.

cp TestLucene90CompoundEntriesJavadoc.java $LUCENE/lucene/core/src/test/org/apache/lucene/codecs/lucene90/
cd $LUCENE && ./gradlew :lucene:core:test --tests org.apache.lucene.codecs.lucene90.TestLucene90CompoundEntriesJavadoc -Ptests.seed=DEADBEEF
# assertions off: add -Ptests.asserts=false (same result)

TestLucene90CompoundEntriesJavadoc > testFileNameIsTheFileName FAILED
    java.lang.AssertionError: FileName values expected:<[_0.fdm, _0.fdt, _0.fdx, _0.fnm]> but was:<[.fdm, .fdt, .fdx, .fnm]>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:F6E6066DC01DD8C4]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:120)
        at org.apache.lucene.codecs.lucene90.TestLucene90CompoundEntriesJavadoc.testFileNameIsTheFileName(TestLucene90CompoundEntriesJavadoc.java:76)

TestLucene90CompoundEntriesJavadoc > testEntryTableEndsAfterLastEntry FAILED
    java.lang.AssertionError: bytes after the documented grammar expected:<0> but was:<16>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:DDD941FE5F8B8BCF]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.apache.lucene.codecs.lucene90.TestLucene90CompoundEntriesJavadoc.testEntryTableEndsAfterLastEntry(TestLucene90CompoundEntriesJavadoc.java:57)

2 tests completed, 2 failed

• The line above. .cfs --> Header, FileData^FileCount, Footer (:49), with its own Footer type line (:57); both files get a footer in the same method (:106-107).
• Convention. Every other explicit whole-file production ((.ext) --> ...) in the core codecs names its Footer, for example the norms metadata and data files (Lucene90NormsFormat.java:46, :71), .tvd/.tvx (Lucene90TermVectorsFormat.java:82, :157), .fnm, .si, the terms dictionary, postings and segments_N; the only other production without it is .liv (LFV-970). Formats documented in prose or as file lists (doc values, points, stored fields, vectors) have no such production to compare.
• History. 4.8.0-4.10.4 listed the Footer on the .cfe line; it was lost when LUCENE-5944 replaced the 4.x text with the trunk text, which had it on the .cfs line instead. No issue or review discusses it.
• FileName. The sentence describes the pre-3.1 writer, which stored the full name; LUCENE-2455 changed the writer and not the sentence.
• The contract. lucene104/package-info.java:50 and :188 (quoted under "Who is affected").

Strongest case for "intended" (made by the refute reviewer, the intended-lens reviewer and the devil's-advocate verifier; the cross-model reviewers rated it moderate): the overview's History says "checksum footers were added to the end of each index file" since 4.8 (package-info.java:381-383), dev-docs/file-formats.md:80-84 says to add headers and footers to all files and TestAllFilesHaveChecksumFooter checks every file, .cfe included, so a per-file grammar need not repeat the footer; the overview calls itself "a high-level definition" (:54); the reader calls the stored string an id (Lucene90CompoundReader.java:144); and CompoundFormat is @lucene.experimental (CompoundFormat.java:28). The .cfs production is not byte-complete either: the 64-byte alignment padding between sub-files is only in the Notes prose. Under the generous footer reading the documented grammar parses all 87 .cfe files of the hunt's sweep.

Why it fails: under that reading the .cfs Footer on the line above would be redundant too, and the explicit whole-file productions in the core codecs do not leave the footer out; the explicit .cfs line invites a literal reading of the .cfe line. "High-level" allows leaving detail to prose, as the .cfs alignment is, not a production that ends 16 bytes early with no prose mentioning those bytes. The id reading conflicts with the plain words "that file's name", which date from when the full name was stored, and the FileName part does not depend on the footer convention at all. @lucene.experimental is about API stability. Committers edited this block twice (LUCENE-10019, #15341) and only touched the alignment note, so that is not an acceptance decision. The generous reading lowers the severity of the footer part; it does not make the text right. The final verdicts of all verifiers and the adjudication kept the defect (the cross-model review with medium confidence), with the intended argument rated moderate by the cross-model reviewers.

Two ways to correct the text; A was validated. Backport: the javadoc lines are the same on branch_10_6 (2 lines higher), so the same text change applies there if wanted; main and branch_11x are the usual targets.

, Footer at the end of the .cfe production; the type line DataOffset,DataLength --> UInt64; FileName --> String, the file name without the segment name (for example .fnm for _0.fnm); and the same words in the Notes sentence. Javadoc only, one file, 7 lines in and 5 out; the Notes sentence is reflowed the way google-java-format wants it.

--- a/lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90CompoundFormat.java
+++ b/lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90CompoundFormat.java
@@ -48,11 +48,12 @@
  * <ul>
  *   <li>Compound (.cfs) --&gt; Header, FileData <sup>FileCount</sup>, Footer
  *   <li>Compound Entry Table (.cfe) --&gt; Header, FileCount, &lt;FileName, DataOffset,
- *       DataLength&gt; <sup>FileCount</sup>
+ *       DataLength&gt; <sup>FileCount</sup>, Footer
  *   <li>Header --&gt; {@link CodecUtil#writeIndexHeader IndexHeader}
  *   <li>FileCount --&gt; {@link DataOutput#writeVInt VInt}
- *   <li>DataOffset,DataLength,Checksum --&gt; {@link DataOutput#writeLong UInt64}
- *   <li>FileName --&gt; {@link DataOutput#writeString String}
+ *   <li>DataOffset,DataLength --&gt; {@link DataOutput#writeLong UInt64}
+ *   <li>FileName --&gt; {@link DataOutput#writeString String}, the file name without the segment
+ *       name (for example <code>.fnm</code> for <code>_0.fnm</code>)
  *   <li>FileData --&gt; raw file data
  *   <li>Footer --&gt; {@link CodecUtil#writeFooter CodecFooter}
  * </ul>
@@ -63,8 +64,9 @@
  *   <li>FileCount indicates how many files are contained in this compound file. The entry table
  *       that follows has that many entries.
  *   <li>Each directory entry contains a long pointer to the start of this file's data section, the
- *       files length, and a String with that file's name. The start of file's data section is
- *       aligned to 64 bytes to not introduce additional unaligned accesses with mmap.
+ *       files length, and a String with that file's name without the segment name. The start of
+ *       file's data section is aligned to 64 bytes to not introduce additional unaligned accesses
+ *       with mmap.
  * </ul>
  */
 public final class Lucene90CompoundFormat extends CompoundFormat {

Trade-offs

• The corrected text was checked against real files: the writer's layout parses all 87 .cfe files of the hunt's 91 generated indexes, all 1720 entries hold the stripped name, and the corrected text, extracted from the patched javadoc, parses all 38 .cfe files of a second 41-configuration sweep.
• Dropping Checksum is cleanup: the linked CodecFooter grammar defines it. Optional extra: the Notes list the entry parts as pointer, length, name, while the grammar order is FileName, DataOffset, DataLength; the sentence could follow the grammar order (not in the diff).

What the checks validated

Gradle in a scratch worktree of 645d4fe14a20 (seed DEADBEEF, 2026-11-01), two runs. Stock: the minimal repro fails 2 of 2 with assertions on and off. Run 2, the diff shown (fix A): the same test following the corrected grammar (CodecUtil.checkFooter after the last entry; names compared with stripSegmentName of listAll()) passes 2 of 2 with assertions on; TestLucene90CompoundFormat passes (36 tests, 0 failures, 1 nightly-only test skipped); :lucene:core:check -x test (google-java-format, ecj lint, forbidden APIs) succeeds. Run 1, on the hunt's text, which has the same grammar lines and differs only in the line breaks of the Notes sentence: the same test also passes with assertions off, TestLucene90CompoundFormat passes and :lucene:core:renderJavadoc renders the corrected grammar, but google-java-format failed on those line breaks, hence the reflowed diff. Option B passes google-java-format (run 2). The verbatim repro keeps failing with the fix, by design: it transcribes the old grammar, and the fix changes no code.

, Footer on the .cfe line and the FileName wording (type line and note), keeping Checksum on the type line.

--- a/lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90CompoundFormat.java
+++ b/lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90CompoundFormat.java
@@ -48,11 +48,12 @@
  * <ul>
  *   <li>Compound (.cfs) --&gt; Header, FileData <sup>FileCount</sup>, Footer
  *   <li>Compound Entry Table (.cfe) --&gt; Header, FileCount, &lt;FileName, DataOffset,
- *       DataLength&gt; <sup>FileCount</sup>
+ *       DataLength&gt; <sup>FileCount</sup>, Footer
  *   <li>Header --&gt; {@link CodecUtil#writeIndexHeader IndexHeader}
  *   <li>FileCount --&gt; {@link DataOutput#writeVInt VInt}
  *   <li>DataOffset,DataLength,Checksum --&gt; {@link DataOutput#writeLong UInt64}
- *   <li>FileName --&gt; {@link DataOutput#writeString String}
+ *   <li>FileName --&gt; {@link DataOutput#writeString String}, the file name without the segment
+ *       name (for example <code>.fnm</code> for <code>_0.fnm</code>)
  *   <li>FileData --&gt; raw file data
  *   <li>Footer --&gt; {@link CodecUtil#writeFooter CodecFooter}
  * </ul>
@@ -63,8 +64,9 @@
  *   <li>FileCount indicates how many files are contained in this compound file. The entry table
  *       that follows has that many entries.
  *   <li>Each directory entry contains a long pointer to the start of this file's data section, the
- *       files length, and a String with that file's name. The start of file's data section is
- *       aligned to 64 bytes to not introduce additional unaligned accesses with mmap.
+ *       files length, and a String with that file's name without the segment name. The start of
+ *       file's data section is aligned to 64 bytes to not introduce additional unaligned accesses
+ *       with mmap.
  * </ul>
  */
 public final class Lucene90CompoundFormat extends CompoundFormat {

Trade-offs

Fixes both wrong statements and leaves the redundant Checksum entry for a reader to puzzle over.

What the checks validated

Not built or tested (the diff applies cleanly on 645d4fe14a20 and passes checkGoogleJavaFormat, run 2).

Leave the .cfe line as is, because the overview says every index file ends with a footer.

Trade-offs

Keeps the .cfe line inconsistent with the .cfs line beside it and does nothing for the FileName wording.

What the checks validated

Not run.

• The check extracts the .cfe and .cfs productions and the shared type list mechanically from the javadoc source, checks them for well-formedness (it reports the unused definition Checksum) and interprets them over the real bytes of each compound segment. The oracle is the segment's CompoundDirectory: entry count, every FileName resolving to an inner file, DataLength equal to the inner file length, 64-byte aligned DataOffset.
• Inputs: 91 indexes (11 fixed and 80 seeded random configurations), 87 compound segments, the same results with assertions on and off.
• Positive control: the writer-layout grammar (the proposed fix text) parses all 87 .cfe files. Vacuity: 3 of 3 layout mutants (footer dropped, DataLength dropped, DataOffset/DataLength swapped) are killed. The .cfs production parses all 87 .cfs files. Fix check: the corrected text parses 38 of 38 files in a second sweep.
• The unit's starting lead was a probe that saw 16 bytes after the documented grammar. This is a differential grammar check of documentation against real output, not model checking or a proof.