The .liv grammar in the Lucene90LiveDocsFormat javadoc lists a Generation field that is not written and leaves out the footer

When a segment has deleted documents, the default codec writes a .liv file with one bit per document. The class javadoc of Lucene90LiveDocsFormat gives its layout as Deletions (.liv) --> IndexHeader,Generation,Bits (Lucene90LiveDocsFormat.java:45 on main 645d4fe14a20), defines SegmentHeader --> IndexHeader (:48), a name the grammar does not use, and leaves LongCount undefined (:49). The writer (:173-182) stores the deletes generation as the IndexHeader's suffix, in base 36, then the bit words, then a 16-byte codec footer; nothing is written between the header and the bits. A decoder that follows the text and reads Generation as a field of its own (the javadoc gives it no type; before LUCENE-5969 it was an Int64) reads the first bit word where it expects the generation (2 instead of 1 in the repro) and finds 16 trailing bytes the grammar does not mention. LUCENE-5969 (2014, before 5.0.0) moved the generation from an Int64 field into the header suffix and kept the word in the grammar; the footer has never been listed. Lucene's reader (:86-98) matches the writer, so only the javadoc is wrong, in every release from 5.0.0 to 10.6.0 and on main.

Deletions (.liv) --> IndexHeader,Generation,Bits

  SegmentHeader --> IndexHeader      (CodecUtil#writeIndexHeader)
  Bits --> <Int64>^LongCount

Generation has no type line, SegmentHeader does not occur in the grammar, LongCount and the bit meaning are not defined, and there is no Footer.

• gen = info.getNextDelGen() (:168); the file name carries it (:169).
• CodecUtil.writeIndexHeader(output, CODEC_NAME, VERSION_CURRENT, info.info.getId(), Long.toString(gen, Character.MAX_RADIX)) (:173-178): the generation is the header suffix, in base 36.
• writeBits(output, bits) (:180): the bits as Int64 words in batches of up to 1024 bits, bits2words(numBitsToCopy) words per batch (:201-213), the bits past maxDoc cleared (:205-207).
• CodecUtil.writeFooter(output) (:182): CodecFooter --> Magic,AlgorithmID,Checksum, 16 bytes (CodecUtil.java:396).
• The reader mirrors it: checkIndexHeader with the same suffix (:86-92), readLongs of bits2words(maxDoc) words (:131-139), checkFooter (:98); a set bit is a live document (:144-145).

IndexHeader (43 bytes) 3f d7 6c 17 10 4c 75 63 65 6e 65 39 30 4c 69 76 65 44 6f 63 73 00 00 00 00
                       68 a3 41 43 0b 4a 55 52 c2 51 b4 61 f2 1a 24 0e 01 31   <- suffix: length 1, "1"
Bits    (8 bytes)      02 00 00 00 00 00 00 00                                  <- Int64 = 2 = binary 10
Footer  (16 bytes)     c0 28 93 e8 00 00 00 00 00 00 00 00 45 f2 e2 c9

The segment id bytes and the checksum differ per run; the layout does not. Whatever integer type a reader assumes for the untyped Generation (Int64, Int32, VLong, VInt), the bytes 02 00 00 ... read as 2. In the hunt's larger index (200 documents, two deletions, 91 bytes) the same reading gives -9 instead of the generation 1, and with Generation skipped the grammar ends at byte 75 of 91, 16 bytes (the footer) short.

15b8b1e66d9 (2014-09-26) added a real output.writeLong(gen) after the header, documented as Generation --> Int64, and the reader compared it with the expected generation ("file mismatch, expected generation=..."). c4bf601b68f (LUCENE-5969, 2014-10-22) moved the generation into the header suffix and removed the write, the reader check and the type line, but kept Generation in the grammar and the label SegmentHeader. The footer was never in the grammar, from the class's first version on.

• Who is hit. Anyone who decodes a .liv file without Lucene, or checks one by hand, following the class javadoc: format inspectors, ports to other languages, recovery or forensic tools. The default codec's overview (lucene104/package-info.java) links this class as Live documents (:177, :303), says it "defines the index file formats used in this version of Lucene" (:50) and "Details on each of these are provided in their linked pages" (:188).
• Default vs opt-in. The default codec, every .liv file it writes, on every release since 5.0.0. No setting changes the layout.
• Scale. No runtime effect. The symptom is confined to a decoder written from the text: a wrong generation right after the header and 16 unexplained trailing bytes. A careful reader who cross-checks CodecUtil and the overview's footer note can work it out, which keeps the impact minor.
• Rating. Impact class minor, likelihood plausible, priority low.

• The test sits in the codec package, like Lucene's own codec tests, and passes the codec name as the literal "Lucene90LiveDocs" (the value of the class's private CODEC_NAME, :58). checkIndexHeader passing with the suffix "1" shows the generation is already in the header; the next read takes Generation as a field of its own, as the grammar lists it. With this input the next bytes are 02 00 00 ..., so any integer type for Generation reads 2; the test uses Int64, the type the element had before LUCENE-5969 (the javadoc itself gives none). It does not check the footer; the hunt's grammar check and the independent reproduction's test do.
• Results: fails 1 of 1 on main with assertions on and off, standalone (1.52 s) and through gradle, and on the published 10.6.0 and 9.12.3 jars; 5 random-seed iterations fail 5 of 5 with the same message. The hunt's and the verifiers' larger tests also fail on 10.5.2 and the branch_10_6 head.
• As a regression test: this test cannot guard the javadoc, because it transcribes the grammar. A form that pins down the corrected layout is this same test with its last line replaced by "next is the Bits word 0b10, then exactly the 16 footer bytes remain" (the fix check below).

cp TestLucene90LiveDocsFormatJavadoc.java $LUCENE/lucene/core/src/test/org/apache/lucene/codecs/lucene90/
cd $LUCENE && ./gradlew :lucene:core:test --tests org.apache.lucene.codecs.lucene90.TestLucene90LiveDocsFormatJavadoc -Ptests.seed=DEADBEEF
# assertions off: add -Ptests.asserts=false (same result)

TestLucene90LiveDocsFormatJavadoc > testGenerationFollowsIndexHeader FAILED
    java.lang.AssertionError: Generation expected:<1> but was:<2>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:5BDBCBCA94AFA0A]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.apache.lucene.codecs.lucene90.TestLucene90LiveDocsFormatJavadoc.testGenerationFollowsIndexHeader(TestLucene90LiveDocsFormatJavadoc.java:60)

1 test completed, 1 failed

• History. 15b8b1e66d9 wrote the generation as an Int64 after the header and documented Generation --> Int64; c4bf601b68f (LUCENE-5969) removed both the write and the type line. History suggests the word left in the grammar is what remains of the deleted field; it does not prove an oversight, because the same author revised the line later (5dcd24bf04c) and kept the word. Either way, the comma-separated grammar now lists an element that has no bytes of its own.
• A half-renamed type list. It defines SegmentHeader, which the grammar no longer uses; no reading makes that definition intended.
• The contract. lucene104/package-info.java:50 and :188 (quoted under "Who is affected").
• Convention. Most core grammars list Footer explicitly: .cfs (Lucene90CompoundFormat.java:49, :57), .nvm/.nvd (Lucene90NormsFormat.java:57, :77), .fnm (Lucene94FieldInfosFormat.java:48-50, :62), .si (Lucene99SegmentInfoFormat.java:67); not all do: the .cfe entry-table production (Lucene90CompoundFormat.java:50-51) does not. A .fnm file written for doc-values updates has a generation suffix in its header (ReadersAndUpdates.java:846, :854), and the .fnm grammar folds it into Header with no Generation element.
• Precedent. File-format doc errors were filed and fixed as ordinary issues, e.g. LUCENE-7532 (#8583) and LUCENE-713 (#1788).

Strongest case for "intended" (the refute reviewer, the intended-lens reviewer and the devil's-advocate verifier; rated moderate): CodecUtil.writeIndexHeader's javadoc gives "generation" as an example suffix (CodecUtil.java:90-92), so IndexHeader,Generation could be read as one element; the overview says every index file ends with a checksum footer (package-info.java:381-383) and calls itself "a high-level definition" (:54), the policy of LUCENE-2946 (#4020); and LiveDocsFormat is @lucene.experimental. Under that reading (Generation zero-width, footer implicit) the bytes parse exactly: 143 of 143 files in the hunt's grammar check.

Why it fails: the grammar is a comma-separated sequence of elements, and Generation was a separate Int64 element until its type line was deleted; a suffixed header elsewhere (.fnm) is documented as plain Header; "high-level" allows leaving detail out, not listing an element that is not written or defining a name the grammar does not use, and on LUCENE-2946 rmuir wrote that bit-for-bit detail is fine "For the simple metadata files, and even stored fields and postings its fine (for now)"; @lucene.experimental is about API stability. The generous reading lowers the severity; it does not make the text right. Every verifier and the adjudication still concluded that the text is wrong as written; the reviewers rated the case for an intended reading moderate.

Two ways to correct the text; A was validated. Backport: the file is byte-identical on branch_10_6, so the same diff applies there if wanted; main and branch_11x are the usual targets.

Grammar IndexHeader,Bits,Footer, the type lines IndexHeader and Footer, and a short Notes list: the header suffix is the deletes generation in base 36 (as in the file name), LongCount = ceil(maxDoc / 64), bit (docID % 64) of word (docID / 64) is set for a live document, and the bits from maxDoc on are clear. Javadoc only.

--- a/lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90LiveDocsFormat.java
+++ b/lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90LiveDocsFormat.java
@@ -42,11 +42,21 @@
  *
  * <p>Although per-segment, this file is maintained exterior to compound segment files.
  *
- * <p>Deletions (.liv) --&gt; IndexHeader,Generation,Bits
+ * <p>Deletions (.liv) --&gt; IndexHeader,Bits,Footer
  *
  * <ul>
- *   <li>SegmentHeader --&gt; {@link CodecUtil#writeIndexHeader IndexHeader}
+ *   <li>IndexHeader --&gt; {@link CodecUtil#writeIndexHeader IndexHeader}
  *   <li>Bits --&gt; &lt;{@link DataOutput#writeLong Int64}&gt; <sup>LongCount</sup>
+ *   <li>Footer --&gt; {@link CodecUtil#writeFooter CodecFooter}
+ * </ul>
+ *
+ * <p>Notes:
+ *
+ * <ul>
+ *   <li>The suffix of the IndexHeader is the deletes generation in base 36, the same generation as
+ *       in the file name.
+ *   <li>LongCount is ceil(maxDoc / 64). Bit (docID % 64) of Int64 number (docID / 64) is set if the
+ *       document is live and clear if it is deleted. The bits from maxDoc on are clear.
  * </ul>
  */
 public final class Lucene90LiveDocsFormat extends LiveDocsFormat {

Trade-offs

• Every added fact was checked on real files: in all 143 .liv files of 91 generated indexes the suffix is the base-36 generation, a set bit is a live document (least significant bit first) and the bits past maxDoc are clear; the writer clears them explicitly (:205-207). The fixed text parses 59 of 59 files in a second sweep.
• 10 more javadoc lines to keep in sync (12 inserted, 2 replaced), on a simple and stable format (VERSION_CURRENT is still VERSION_START).

What the checks validated

• Scratch worktree of 645d4fe14a20, gradle, seed DEADBEEF (2026-11-01): on stock main the minimal test fails with assertions on and off.
• With fix A applied (2 lines changed, 10 added; 12 insertions, 2 deletions): the same test with its last line following the corrected grammar (next the Bits word 0b10, then exactly the 16 footer bytes remain) passes 1 of 1 with assertions on and off; TestLucene90LiveDocsFormat passes (4 tests, 0 failures, testOverflow skipped as a monster test); :lucene:core:check -x test (google-java-format, ecj lint, forbidden APIs) passes; :lucene:core:renderJavadoc passes and renders Deletions (.liv) --> IndexHeader,Bits,Footer with the type lines and notes. A javadoc edit cannot change the bytes, so the corrected-grammar test checks that the new text describes them, not a code change.
• Earlier: the refute reviewer applied the same text to a scratch copy (live-docs tests and javadoc lint passed), and the hunt's grammar check parses 59 of 59 .liv files with it.

Grammar IndexHeader,Bits,Footer, the label SegmentHeader changed to IndexHeader, and a Footer --> CodecFooter type line; no notes.

- * <p>Deletions (.liv) --&gt; IndexHeader,Generation,Bits
+ * <p>Deletions (.liv) --&gt; IndexHeader,Bits,Footer
  *
  * <ul>
- *   <li>SegmentHeader --&gt; {@link CodecUtil#writeIndexHeader IndexHeader}
+ *   <li>IndexHeader --&gt; {@link CodecUtil#writeIndexHeader IndexHeader}
  *   <li>Bits --&gt; &lt;{@link DataOutput#writeLong Int64}&gt; <sup>LongCount</sup>
+ *   <li>Footer --&gt; {@link CodecUtil#writeFooter CodecFooter}
  * </ul>

Trade-offs

Removes the wrong element and the wrong label and lists the footer, but a reader still has to find out from the code that the header suffix holds the generation, what LongCount is and which bit value means live.

What the checks validated

Not built (the diff applies cleanly to 645d4fe14a20, git apply --check).

For example "Generation: the deletes generation, stored as the IndexHeader suffix".

Trade-offs

Keeps a grammar element that has no bytes of its own, unlike the .fnm convention (Header only), and still needs the label and Footer fixes.

What the checks validated

Not run.

• The check extracts the production and type definitions mechanically from the javadoc source, checks them for well-formedness (it reports the undefined element Generation, the undefined count LongCount and the unused definition SegmentHeader) and interprets them over the real bytes, under every typing of the untyped Generation: absent, Int64, Int32, VLong, VInt.
• Inputs: 91 indexes (11 fixed and 80 seeded random configurations: sizes, flushes, compound files, deletion rates for both live-docs read paths, 1-3 deletion commits, many field types), 143 .liv files, the same results with assertions on and off. LiveDocsFormat.readLiveDocs is the oracle for every bit.
• Positive control: the writer-layout grammar parses all 143 files exactly. Vacuity: 3 of 3 .liv layout mutants (footer dropped, Int64 generation inserted, LongCount + 1) are killed. Fix check: the corrected text parses 59 of 59 files.
• This is a differential grammar check of documentation against real output, not model checking or a proof.