Lucene99SegmentInfoFormat's javadoc gives a wrong .si layout: the segment version is three Int32s, not a String

The default codec writes each segment's .si file with Lucene99SegmentInfoFormat, and the file-format overview links that class as the Segment Info format and sends readers there for the details. Its javadoc gives the file as Header, SegVersion, SegSize, IsCompoundFile, Diagnostics, Files, Attributes, IndexSort, Footer and types SegVersion and SegMinVersion as DataOutput#writeString Strings. Since Lucene 5.0 the writer stores the version as three Int32s (major, minor, bugfix), then a flag byte and, when it is 1, the minimum version as three more Int32s, then SegSize, IsCompoundFile and a HasBlocks byte. A parser that follows the javadoc reads the first Int32 (major version 11, 0B 00 00 00) as a string length and gets 11 NUL characters instead of 11.0.0; a 1-document segment then reads as SegSize 2,817, and every later field is misaligned. Lucene's own reader matches its writer, so Lucene reads its own files correctly and no runtime effect is expected. The text was correct when written (2012) and went stale in three steps: LUCENE-5992 (5.0.0), LUCENE-7756 (7.0.0, SegMinVersion) and #12685 (9.9.0, HasBlocks). A javadoc-only fix was validated.

The reader, parseSegmentInfo (:121-179), mirrors the writer, so Lucene reads its own files correctly.

After the 45-byte header the file starts 0B 00 00 00 | 00 00 00 00 | 00 00 00 00 | 01 | 0B 00 00 00 | 00 00 00 00 | 00 00 00 00: version 11.0.0, flag 1, minimum version 11.0.0 (a flushed segment records the current version for both).

• readString() at SegVersion reads the length byte 0x0B = 11 and then 11 bytes: the rest of the major version and all of minor and bugfix, all zero. It returns 11 NUL characters and stops exactly at the flag byte.
• readInt() for SegSize then reads 01 0B 00 00 = 2817 (the segment has 1 document); the next byte, 0, is taken as IsCompoundFile (the segment is a compound file, 1); the following zero byte reads as an empty Diagnostics map (the real one has 8 entries). Every later field is garbage.
• The number of garbage characters is the major version: 10 on 10.6.0, 9 on 9.9.0. Before 9.0 an Int32 was big-endian (00 00 00 05), so the documented read returned an empty string.

No runtime, index, merge or search effect. The other .si elements (SegSize, IsCompoundFile, Diagnostics, Files, Attributes, IndexSort) are placed and typed consistently with the bytes, apart from the IndexSort --> writeVInt Int32 count wording (:63), which is cosmetic (it is a VInt). The compound-file value and the 8 diagnostics entries above are properties of this one-document configuration, not of the format.

• Who is hit. Developers who decode .si files themselves from the javadoc: ports of Lucene to other languages, index inspection or recovery tools, format tooling. The file-format overview calls itself "a high-level definition" (lucene104/package-info.java:54) and sends readers to the linked class pages for the details (:188); for .si that page is this javadoc (:232).
• Default vs opt-in. The default codec (Lucene104Codec.java:73): every segment it writes has a .si file and the mismatch is in its first field. No setting, input or scale is needed. Other codecs are outside this claim.
• Who is not hit. Applications that read and write indexes through Lucene: the supported reader, SegmentInfoFormat.read, matches the writer, and TestLucene99SegmentInfoFormat (17 tests) passes.
• How long. From 5.0.0 to 10.6.0 and main, shown by representative runs per format class and the source history, not a run on every patch release; Lucene99SegmentInfoFormat itself dates from 9.9.0, earlier releases carry predecessor classes with the same text. The recorded searches found no public report; it is rated minor and low priority.

• The test pins the codec with TestUtil.getDefaultCodec() (the test framework otherwise randomizes it) and uses a ByteBuffersDirectory. "Lucene90SegmentInfo" is the codec name the class writes into the header (Lucene99SegmentInfoFormat.java:94). The message prints the char codes because NUL characters are invisible.
• It asserts only the first divergence. The missing SegMinVersion and HasBlocks are visible by reading the javadoc next to writeSegmentInfo; the byte walk in the explanation above shows what the rest of a documented parse reads.
• Results: fails 1 of 1 on main with assertions on and off, standalone and through gradle; on the published 10.6.0 jars (SegVersion chars [0, 0, 0, 6, 0, 0, 0, 0, 0, 0] expected:<[10.6.0]>) and 9.9.0 jars ([0, 0, 0, 9, 0, 0, 0, 0, 0] expected:<[9.9.0]>).
• As a regression test: the code is covered by TestLucene99SegmentInfoFormat, and this test encodes the old text, so it would not guard the new one.

cp TestLucene99SegmentInfoFormatJavadoc.java $LUCENE/lucene/core/src/test/org/apache/lucene/codecs/lucene99/
cd $LUCENE && ./gradlew :lucene:core:test --tests org.apache.lucene.codecs.lucene99.TestLucene99SegmentInfoFormatJavadoc -Ptests.seed=DEADBEEF
# assertions off: add -Ptests.asserts=false (same result)

# excerpt: gradle on main 645d4fe14a20, -Ptests.seed=DEADBEEF; with -Ptests.asserts=false the lines are identical
TestLucene99SegmentInfoFormatJavadoc > testSegVersionIsTheDocumentedString FAILED
    org.junit.ComparisonFailure: SegVersion chars [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] expected:<[11.0.0]> but was:<[<11 NUL characters>]>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:83CC134AEA3AF546]:0)
        at app//org.junit.Assert.assertEquals(Assert.java:117)
        at app//org.apache.lucene.codecs.lucene99.TestLucene99SegmentInfoFormatJavadoc.testSegVersionIsTheDocumentedString(TestLucene99SegmentInfoFormatJavadoc.java:45)
:lucene:core:test (FAILURE): 1 test, 1  failure
# (excerpt: the suite-output lines and the repeated failure lines before the summary are left out;
#  the 11 NUL characters are printed raw by JUnit and shown here as <11 NUL characters>)

• Contract text. Lucene99SegmentInfoFormat.java:48-67 lists the elements in order and links each type to its DataOutput writer method (SegVersion --> {@link DataOutput#writeString String}); nothing marks it as approximate. lucene104/package-info.java:188: "Details on each of these are provided in their linked pages."
• History. LUCENE-4055 (f330cf65adc, Robert Muir, 2012-05-22) wrote "SegVersion --> String" when the writer wrote the version with writeString (output.writeString(si.getVersion())); 4.10.4 still writes writeString(version.toString()), and the documented grammar parses there. LUCENE-5992 (#7054, a922e891914 / 776682ba9c3, Michael McCandless, 2014-10-06, "encode version using 3 ints, not String") changed the writer and reader, not the javadoc; its 22 comments never mention it (in review uschindler noticed that a different javadoc, Version#write, had become wrong). LUCENE-7756 (23b002a0fdf, 2017) and #12685 (6677109ee60, 2023) added SegMinVersion and HasBlocks to the type list and descriptions but not to the sequence; their reviews do not discuss it.
• Consistency. SegmentInfos.java:73-77 documents the commit's Lucene version as "three {@link DataOutput#writeVInt vInt}: major, minor, bugfix", and the hunt found those fields match the bytes.
• Precedent. LUCENE-713, LUCENE-7532 (#8583; fix commit 830a3fdfbf1, "add back lost codec file format documentation") and LUCENE-9353 (#90) fixed or restored file-format documentation.
• No waiver. No test, CHANGES or MIGRATE entry, review comment or list thread accepts the stale text.

Strongest case for "intended" (the refute reviewer and the cross-model review rated it moderate; the intended-behaviour check set it out as the strongest case and rejected it): the format classes are @lucene.experimental; the overview says it "attempts to provide a high-level definition of the Apache Lucene file formats", a policy from LUCENE-2946 (2012, "change file format documentation from bit-for-bit to highlevel"); the supported reader is SegmentInfoFormat.read; and the round trip is tested.

Why it fails: a wrong primitive type (String versus three Int32s) and two elements missing from an ordered list are not a high-level simplification. In the LUCENE-2946 thread (#4020, 2012-04-25) Robert Muir wrote "we can go into detail, but we can't do bit-for-bit with even StandardCodec" and "For the simple metadata files, and even stored fields and postings its fine (for now), but e.g. going bit-for-bit with packed integer compression of docvalues isnt very realistic" (their words), and .si is such a simple metadata file. @lucene.experimental allows the format to change, not its description to be wrong. Every verifier and the adjudication called it real, and the audit confirmed it.

A is the fix; B to D are optional additions. Upstream retired branch_10x on 2026-10-29 (#16911), so the targets are main and branch_11x, plus branch_10_6 if wanted (its copy of the class carries the same lines).

Javadoc only, Lucene99SegmentInfoFormat.java:48-79, 9 lines added, 6 removed: place SegMinVersion and HasBlocks in the sequence, type SegVersion as three Int32s and SegMinVersion as an Int8 flag plus three Int32s, say that the flag is 0 when the minimum version is unknown (as SegmentInfo.getMinVersion() documents), and give the HasBlocks values (-1 or 1, like IsCompoundFile).

diff --git a/lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99SegmentInfoFormat.java b/lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99SegmentInfoFormat.java
index 8512307156a..e2ee40d2b5f 100644
--- a/lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99SegmentInfoFormat.java
+++ b/lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99SegmentInfoFormat.java
@@ -45,8 +45,8 @@ import org.apache.lucene.util.Version;
  * <p>Files:
  *
  * <ul>
- *   <li><code>.si</code>: Header, SegVersion, SegSize, IsCompoundFile, Diagnostics, Files,
- *       Attributes, IndexSort, Footer
+ *   <li><code>.si</code>: Header, SegVersion, SegMinVersion, SegSize, IsCompoundFile, HasBlocks,
+ *       Diagnostics, Files, Attributes, IndexSort, Footer
  * </ul>
  *
  * Data types:
@@ -54,8 +54,9 @@ import org.apache.lucene.util.Version;
  * <ul>
  *   <li>Header --&gt; {@link CodecUtil#writeIndexHeader IndexHeader}
  *   <li>SegSize --&gt; {@link DataOutput#writeInt Int32}
- *   <li>SegVersion --&gt; {@link DataOutput#writeString String}
- *   <li>SegMinVersion --&gt; {@link DataOutput#writeString String}
+ *   <li>SegVersion --&gt; three {@link DataOutput#writeInt Int32}: major, minor, bugfix
+ *   <li>SegMinVersion --&gt; {@link DataOutput#writeByte Int8} flag, 0 or 1, followed, if it is 1,
+ *       by three {@link DataOutput#writeInt Int32}: major, minor, bugfix
  *   <li>Files --&gt; {@link DataOutput#writeSetOfStrings Set&lt;String&gt;}
  *   <li>Diagnostics,Attributes --&gt; {@link DataOutput#writeMapOfStrings Map&lt;String,String&gt;}
  *   <li>IsCompoundFile --&gt; {@link DataOutput#writeByte Int8}
@@ -71,12 +72,14 @@ import org.apache.lucene.util.Version;
  *
  * <ul>
  *   <li>SegVersion is the code version that created the segment.
- *   <li>SegMinVersion is the minimum code version that contributed documents to the segment.
+ *   <li>SegMinVersion is the minimum code version that contributed documents to the segment. Its
+ *       flag is 0 if that version is unknown.
  *   <li>SegSize is the number of documents contained in the segment index.
  *   <li>IsCompoundFile records whether the segment is written as a compound file or not. If this is
  *       -1, the segment is not a compound file. If it is 1, the segment is a compound file.
  *   <li>HasBlocks records whether the segment contains documents written as a block and guarantees
- *       consecutive document ids for all documents in the block
+ *       consecutive document ids for all documents in the block. If this is -1, the segment has no
+ *       such documents. If it is 1, it has.
  *   <li>The Diagnostics Map is privately written by {@link IndexWriter}, as a debugging aid, for
  *       each segment it creates. It includes metadata like the current Lucene version, OS, Java
  *       version, why the segment was created (merge, flush, addIndexes), etc.

Trade-offs

• None for users: documentation only; no CHANGES entry strictly needed.
• The sequence and types are what the refute reviewer, the independent reproduction and the adjudication proposed; the wording is this doc's.

What the checks validated

• The new text matches the bytes of the recorded configuration. A fix-text check written only from the patched javadoc (Header; three Int32 for SegVersion; the flag and three Int32 for SegMinVersion; SegSize; the IsCompoundFile and HasBlocks bytes; Diagnostics, Files, Attributes; the sort count) compares every field with SegmentInfo and ends exactly at the footer: passes 1 of 1 on main with assertions on and off and on the 10.6.0 jars. Scope: one flushed one-document segment, min version present (flag 1), no index sort; it does not exercise flag 0, sort fields or the reader's rejection of other flag values. It is a validation, not a proposed test.
• Build checks on the patched tree (scratch worktree of 645d4fe14a20): ./gradlew :lucene:core:test --tests "org.apache.lucene.codecs.lucene99.TestLucene99SegmentInfoFormat" plus the fix-text check, -Ptests.seed=DEADBEEF: 18 tests, 0 failures; :lucene:core:checkGoogleJavaFormat and :lucene:core:renderJavadoc pass, and the rendered page shows the new sequence and types and no SegVersion --> String.

backward_codecs/lucene90/Lucene90SegmentInfoFormat.java:46-47 and :55-56 carry the same stale sequence and String types, and its test writer Lucene90RWSegmentInfoFormat.java:48, :57-58 copies them. That format has no HasBlocks: its reader (:118-142) reads the version, the min-version flag, the doc count, the compound byte, then the maps.

Trade-offs

Harmless, but some committers prefer not to touch frozen formats.

What the checks validated

Not run.

The same hunt found stale grammars in other default-codec file-format javadocs (.fnm, segments_N, .nvm, block-tree terms metadata, vector formats; separate findings and fix sites), which could be fixed in the same change, as could the cosmetic IndexSort --> writeVInt Int32 count wording.

Trade-offs

A bigger change to review; each part is still javadoc only.

#13342 ("Datacube format changes to support materialized views", open, last updated 2025-09-30) adds a Lucene910SegmentInfoFormat whose javadoc copies the stale parts verbatim: the same .si sequence and the same SegVersion and SegMinVersion String lines (it changes the class lead text and adds DataCubesConfig and DataCubeField entries). If it lands, its copy needs the same fix.

Trade-offs

Outside this fix site; it matters only if #13342 lands.

• Each grammar is data in a small grammar language; one interpreter parses the bytes under the documented and the written grammar and compares every value with the real reader.
• 64 default-codec indexes (3 fixed, 1 dense-vector and 60 seeded random feature configurations), 1,114 file checks over 11 file types, including .si.
• Positive control: the writer grammars parse every file exactly to the footer. Vacuity check: 14 of 14 writer-grammar mutants with one element dropped are caught.
• JUnit tests on public Lucene APIs then bridged the finding to the stock jars (later reduced to the one-test repro), and per-release ports dated each of the three stale parts.
• This is a differential check of documentation against bytes, not model checking or a proof.