The default codec's .fnm file-format javadoc gives a wrong field grammar: it leaves out the IndexOptions byte and later elements, and its tables miss codes Lucene writes

The class javadoc of Lucene94FieldInfosFormat, the field infos format of the default codec (Lucene104Codec.java:72), gives the .fnm file as Header, FieldsCount, <FieldName, FieldNumber, FieldBits, DocValuesBits, DocValuesGen, Attributes, DimensionCount, DimensionNumBytes>^FieldsCount, Footer (Lucene94FieldInfosFormat.java:48-50 on main 645d4fe14a20). The writer (:431-456) writes, per field, FieldName, FieldNumber, FieldBits, IndexOptions, a doc-values type byte, a doc-values skip-index byte, DocValuesGen, Attributes, PointDimensionCount, PointIndexDimensionCount and PointNumBytes only when the dimension count is non-zero, then VectorDimension, VectorEncoding and VectorSimilarityFunction; the reader (:147-213) reads the same order.

IndexOptions is typed as a Byte (:58) but never placed; the skip-index byte and PointIndexDimensionCount appear nowhere; the vector elements are described (:104-118) but not placed. The tables miss written values: DocValuesBits is described as a norms nibble plus a doc-values nibble with codes 0..3 (:88-97), but the byte holds one code, and SORTED_SET is 4 and SORTED_NUMERIC 5; FieldBits lists 0x1, 0x2 and 0x4 but 0x8 (soft-deletes field) and 0x10 (parent field) are set; IndexOptions 5 (since 10.5.0) and VectorEncoding 2 (FLOAT16, main only) are missing.

A parser that follows the javadoc reads a text field's IndexOptions byte (3) as "SortedDocValues" and is misaligned from the first field on. The sequence has been wrong since 5.0.0: LUCENE-6013 moved IndexOptions into its own byte and updated the type list, not the sequence. Later changes edited parts of the javadoc (6.0.0 added the point elements to the sequence, 9.0.0 and 9.4.0 described the vector elements) but none placed IndexOptions or brought the sequence and the tables up to date. Documentation only; the fix is javadoc text.

FieldInfos (.fnm) --> Header,FieldsCount, <FieldName,FieldNumber,
FieldBits,DocValuesBits,DocValuesGen,Attributes,DimensionCount,DimensionNumBytes>
FieldsCount,Footer

For one TextField f = "a", the field record is 01 66 | 00 | 00 | 03 | 00 | 00 | ff ff ff ff ff ff ff ff | 02 ...(2 attributes)... | 00 | 00 | 01 | 00 (dumped by the fix's validation check, which parses it with the corrected grammar exactly to the footer):

• Lucene reads: FieldName "f", FieldNumber 0, FieldBits 0, IndexOptions 3 (DOCS_AND_FREQS_AND_POSITIONS), doc values 0, skip index 0, DocValuesGen -1, 2 attributes (the per-field postings format and suffix), PointDimensionCount 0, VectorDimension 0, VectorEncoding 1 (FLOAT32), similarity 0 (EUCLIDEAN).
• The javadoc's order takes 03 as DocValuesBits (low nibble 3 = SortedDocValues), the next 8 bytes, 00 00 ff ff ff ff ff ff, as DocValuesGen (-65536, not -1), and ff ff 02 as the attribute count (49151, not 2); reading 49151 map entries then throws java.io.EOFException at the end of the 153-byte file (the diagram's probe, assertions on and off). An independent verifier's literal-sequence test also measured -65536 for point and vector fields.

The first line says "Lucene 9.0 Field Infos format." (:44); the field descriptions call the point elements PointDimensionCount and PointNumBytes (:102) while the sequence says DimensionCount and DimensionNumBytes; and the write() comment "pack the DV type and hasNorms in one byte" (:444) has been wrong since LUCENE-6006 (5.0.0).

• Who is hit. Anyone who decodes .fnm from the class javadoc instead of from read(): ports of Lucene to other languages, index inspectors, recovery or forensic tools, and developers learning the format.
• Default vs opt-in. The default codec: Lucene104Codec installs this format (Lucene104Codec.java:72), and the file-format overview links this class for the .fnm file (lucene104/package-info.java:147, :243). Every field record carries the IndexOptions byte, so the mismatch shows on the first field of every default-codec .fnm with at least one field, whatever the field (a segment with no fields has no field record to misread).
• Scale. A parser built from the sequence misreads the first field and every later one; one built from the tables rejects or misreads valid codes (any SORTED_SET or SORTED_NUMERIC field, any index with a soft-deletes or parent field). The hunt's grammar check found the documented grammar diverging on 125 of 125 .fnm files from 64 generated default-codec indexes, while the writer's layout parsed all of them to the footer.
• What it does not do. No wrong results, no index change, no runtime path: Lucene's writer and reader agree, and TestLucene94FieldInfosFormat passes. No earlier report was found. Impact class minor, priority low.

• The test reads the header with CodecUtil.checkIndexHeader and then the documented elements one by one. The failing byte is written at Lucene94FieldInfosFormat.java:442 (indexOptionsByte), right before the real doc-values byte at :445.
• Results: fails 1 of 1 on main with assertions on and off (seed DEADBEEF; seed 5EED1234 gives the same message; the test has no random input), standalone and through gradle. The hunt's larger tests, which also check the value tables, fail on main, the 10.6.0 release commit and 10.5.2. On 9.12.3 only its sequence test is evidence (it fails): its value-table test assumes the 10.0 skip-index byte and ends in an EOFException there.
• Even simpler for a reader: put the sequence line (:48-50) next to write() (:431-456).
• As a regression test: this test would not guard the javadoc. It transcribes today's documented order, and a doc fix leaves the bytes unchanged, so a regression form would read the corrected order (as the fix's validation check does).

cp TestLucene94FieldInfosFormatJavadoc.java $LUCENE/lucene/core/src/test/org/apache/lucene/codecs/lucene94/
cd $LUCENE && ./gradlew :lucene:core:test --tests org.apache.lucene.codecs.lucene94.TestLucene94FieldInfosFormatJavadoc -Ptests.seed=DEADBEEF
# assertions off: add -Ptests.asserts=false (same result)

TestLucene94FieldInfosFormatJavadoc > testDocumentedFieldSequence FAILED
    java.lang.AssertionError: DocValuesBits, DocValues nibble expected:<0> but was:<3>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:AF4ECEFFEFFD713C]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.apache.lucene.codecs.lucene94.TestLucene94FieldInfosFormatJavadoc.testDocumentedFieldSequence(TestLucene94FieldInfosFormatJavadoc.java:52)

• Contract text. Lucene94FieldInfosFormat.java:48-62: production notation with every element typed and linked to a DataOutput method. lucene104/package-info.java:50: "This document defines the index file formats used in this version of Lucene"; :188: "Details on each of these are provided in their linked pages."
• History. In LUCENE-6013 (#7075, 2014) the author wrote "I'll update the javadocs for the format changes" and then "New patch, fixing docs"; the type list and the tables were updated, the sequence line was not, and none of the saved issue comments discusses the sequence, so this reads as an omission, not a decision. Later changes that added bytes or values changed the writer and reader together; some edited the javadoc (LUCENE-6861 put the point elements into the sequence, LUCENE-9322 and LUCENE-10577 described the vector elements), but none placed IndexOptions or the newer bytes, and none of their saved PR discussions mentions the stale sequence.
• Tests. No test compares a format javadoc with the bytes. TestLucene94FieldInfosFormat tests codec behaviour (the BaseFieldInfoFormatTestCase round trips, and testVectorSimilarityFuncs checks SIMILARITY_FUNCTIONS directly) but never compares the javadoc grammar with the written bytes.
• Precedent. Upstream has fixed file-format documentation errors of this kind before: LUCENE-713 (#1788), LUCENE-3072 (#4145), LUCENE-7532 (#8583, "Add back lost codec file format documentation").

Strongest case for "intended" (the refute reviewer, the intended-lens cross-model reviewer and the devil's-advocate verifier; rated moderate): the format classes are @lucene.experimental (:121); LUCENE-2946 (#4020, 2012) moved the file-format docs from "bit-for-bit" to "high-level", and package-info.java:54 still says "This document attempts to provide a high-level definition"; the supported way to read .fnm is FieldInfosFormat.read; and the runtime round trip is tested.

Why it fails: in the same LUCENE-2946 thread rmuir wrote (2012-04-25) "we can go into detail, but we can't do bit-for-bit with even StandardCodec... For the simple metadata files, and even stored fields and postings its fine (for now)", and .fnm is such a simple metadata file. The page is presented in exact-looking production notation; its tables are wrong rather than abbreviated; and @lucene.experimental allows the format to change, not the current format to be described wrongly. Every verifier and the adjudication rejected the "intended" reading.

Three ways to resolve it; A was validated. Targets: main and branch_11x, plus branch_10_6 if wanted (branch_10x was decommissioned, #16911). The file is byte-identical on all of them and on the 10.6.0 tag, so the diff applies unchanged; for branch_10_6 drop the FLOAT16 line, since 10.6.0's VectorEncoding has no FLOAT16; the format-version paragraph applies there unchanged (10.6.0 also writes version 2 and reads 0 and 1).

Optional extras for the same change: the write() comment "pack the DV type and hasNorms in one byte" (:444, a code comment, not in the diff), and, on main, the frozen backward-codecs Lucene90FieldInfosFormat, which carries the same stale text (branch_10_6 also has Lucene60FieldInfosFormat).

The new sequence is <FieldName,FieldNumber,FieldBits,IndexOptions,DocValuesType,DocValuesSkipIndexType,DocValuesGen,Attributes,PointDimensionCount,PointIndexDimensionCount?,PointNumBytes?,VectorDimension,VectorEncoding,VectorSimilarityFunction>^FieldsCount. The type list names every byte element and links the VInt ones to writeVInt; the tables gain FieldBits 0x8 and 0x10 (linked to IndexWriterConfig#setSoftDeletesField and #setParentField), IndexOptions 5, a single-code doc-values table 0..5, a skip-index table (0 NONE, 1 RANGE) and VectorEncoding 2; IndexOptions 1 is named DOCS, the current enum constant, not DOCS_ONLY; a new paragraph says this is format version 2 (written since 10.0) and that read() also accepts version 1 (9.10-9.12, no DocValuesSkipIndexType byte) and version 0 (9.4-9.9, also never FieldBits 0x10); the first line becomes "Lucene 9.4 Field Infos format.". One file, +32/-14 lines.

diff --git a/lucene/core/src/java/org/apache/lucene/codecs/lucene94/Lucene94FieldInfosFormat.java b/lucene/core/src/java/org/apache/lucene/codecs/lucene94/Lucene94FieldInfosFormat.java
index acd4bf55152..b0c37f52dd7 100644
--- a/lucene/core/src/java/org/apache/lucene/codecs/lucene94/Lucene94FieldInfosFormat.java
+++ b/lucene/core/src/java/org/apache/lucene/codecs/lucene94/Lucene94FieldInfosFormat.java
@@ -41,13 +41,18 @@ import org.apache.lucene.store.IndexInput;
 import org.apache.lucene.store.IndexOutput;
 
 /**
- * Lucene 9.0 Field Infos format.
+ * Lucene 9.4 Field Infos format.
  *
  * <p>Field names are stored in the field info file, with suffix <code>.fnm</code>.
  *
  * <p>FieldInfos (.fnm) --&gt; Header,FieldsCount, &lt;FieldName,FieldNumber,
- * FieldBits,DocValuesBits,DocValuesGen,Attributes,DimensionCount,DimensionNumBytes&gt;
- * <sup>FieldsCount</sup>,Footer
+ * FieldBits,IndexOptions,DocValuesType,DocValuesSkipIndexType,DocValuesGen,Attributes,
+ * PointDimensionCount,PointIndexDimensionCount?,PointNumBytes?,
+ * VectorDimension,VectorEncoding,VectorSimilarityFunction&gt; <sup>FieldsCount</sup>,Footer
+ *
+ * <p>This is format version 2, written since Lucene 10.0. The reader also accepts version 1 (Lucene
+ * 9.10 to 9.12), which has no DocValuesSkipIndexType byte, and version 0 (Lucene 9.4 to 9.9), which
+ * in addition never sets FieldBits 0x10.
  *
  * <p>Data types:
  *
@@ -55,8 +60,10 @@ import org.apache.lucene.store.IndexOutput;
  *   <li>Header --&gt; {@link CodecUtil#checkIndexHeader IndexHeader}
  *   <li>FieldsCount --&gt; {@link DataOutput#writeVInt VInt}
  *   <li>FieldName --&gt; {@link DataOutput#writeString String}
- *   <li>FieldBits, IndexOptions, DocValuesBits --&gt; {@link DataOutput#writeByte Byte}
- *   <li>FieldNumber, DimensionCount, DimensionNumBytes --&gt; {@link DataOutput#writeInt VInt}
+ *   <li>FieldBits, IndexOptions, DocValuesType, DocValuesSkipIndexType, VectorEncoding,
+ *       VectorSimilarityFunction --&gt; {@link DataOutput#writeByte Byte}
+ *   <li>FieldNumber, PointDimensionCount, PointIndexDimensionCount, PointNumBytes, VectorDimension
+ *       --&gt; {@link DataOutput#writeVInt VInt}
  *   <li>Attributes --&gt; {@link DataOutput#writeMapOfStrings Map&lt;String,String&gt;}
  *   <li>DocValuesGen --&gt; {@link DataOutput#writeLong(long) Int64}
  *   <li>Footer --&gt; {@link CodecUtil#writeFooter CodecFooter}
@@ -76,36 +83,47 @@ import org.apache.lucene.store.IndexOutput;
  *         <li>If the second lowest order-bit is set (0x2), norms are omitted for the indexed field.
  *         <li>If the third lowest-order bit is set (0x4), payloads are stored for the indexed
  *             field.
+ *         <li>If the fourth lowest-order bit is set (0x8), the field is the soft-deletes field (see
+ *             {@link org.apache.lucene.index.IndexWriterConfig#setSoftDeletesField}).
+ *         <li>If the fifth lowest-order bit is set (0x10), the field is the parent field (see
+ *             {@link org.apache.lucene.index.IndexWriterConfig#setParentField}).
  *       </ul>
  *   <li>IndexOptions: a byte containing index options.
  *       <ul>
  *         <li>0: not indexed
- *         <li>1: indexed as DOCS_ONLY
+ *         <li>1: indexed as DOCS
  *         <li>2: indexed as DOCS_AND_FREQS
  *         <li>3: indexed as DOCS_AND_FREQS_AND_POSITIONS
  *         <li>4: indexed as DOCS_AND_FREQS_AND_POSITIONS_AND_OFFSETS
+ *         <li>5: indexed as DOCS_AND_CUSTOM_FREQS
  *       </ul>
- *   <li>DocValuesBits: a byte containing per-document value types. The type recorded as two
- *       four-bit integers, with the high-order bits representing <code>norms</code> options, and
- *       the low-order bits representing {@code DocValues} options. Each four-bit integer can be
- *       decoded as such:
+ *   <li>DocValuesType: a byte containing the field's doc values type:
  *       <ul>
  *         <li>0: no DocValues for this field.
  *         <li>1: NumericDocValues. ({@link DocValuesType#NUMERIC})
- *         <li>2: BinaryDocValues. ({@code DocValuesType#BINARY})
- *         <li>3: SortedDocValues. ({@code DocValuesType#SORTED})
+ *         <li>2: BinaryDocValues. ({@link DocValuesType#BINARY})
+ *         <li>3: SortedDocValues. ({@link DocValuesType#SORTED})
+ *         <li>4: SortedSetDocValues. ({@link DocValuesType#SORTED_SET})
+ *         <li>5: SortedNumericDocValues. ({@link DocValuesType#SORTED_NUMERIC})
+ *       </ul>
+ *   <li>DocValuesSkipIndexType: a byte containing the type of the doc values skip index:
+ *       <ul>
+ *         <li>0: no skip index. ({@link DocValuesSkipIndexType#NONE})
+ *         <li>1: a skip index over value ranges. ({@link DocValuesSkipIndexType#RANGE})
  *       </ul>
  *   <li>DocValuesGen is the generation count of the field's DocValues. If this is -1, there are no
  *       DocValues updates to that field. Anything above zero means there are updates stored by
  *       {@link DocValuesFormat}.
  *   <li>Attributes: a key-value map of codec-private attributes.
- *   <li>PointDimensionCount, PointNumBytes: these are non-zero only if the field is indexed as
- *       points, e.g. using {@link org.apache.lucene.document.LongPoint}
+ *   <li>PointDimensionCount: it is non-zero only if the field is indexed as points, e.g. using
+ *       {@link org.apache.lucene.document.LongPoint}. PointIndexDimensionCount and PointNumBytes
+ *       follow only if it is non-zero.
  *   <li>VectorDimension: it is non-zero if the field is indexed as vectors.
  *   <li>VectorEncoding: a byte containing the encoding of vector values:
  *       <ul>
  *         <li>0: BYTE. Samples are stored as signed bytes
  *         <li>1: FLOAT32. Samples are stored in IEEE 32-bit floating point format.
+ *         <li>2: FLOAT16. Samples are stored in IEEE 16-bit floating point format.
  *       </ul>
  *   <li>VectorSimilarityFunction: a byte containing distance function used for similarity
  *       calculation.

Trade-offs

• The ? marks for the two conditional point elements follow the notation the default postings format javadoc already uses (Lucene104PostingsFormat.java:104); the description says they follow only when PointDimensionCount is non-zero.
• Renaming DocValuesBits to DocValuesType is the one cosmetic choice; keeping the old name gives a smaller diff, but the text must still say the byte is a single code.
• DOCVALUES_SKIPPER = 0x20 (:479, a leftover of #13449 that #13784 replaced before any release) is never written, so the text leaves it out.
• The grammar is the current writer layout (format version 2, FORMAT_CURRENT at :467-471, written at :424). read() also accepts versions 0 and 1 (:138-145): it reads the skip-index byte only from version 2 (:189-194) and rejects the 0x10 bit below version 1 (:165-176); FORMAT_CURRENT is 0 at the 9.4.0-9.9.2 tags and 1 at 9.10.0-9.12.3. The fix states this in one paragraph; the validation parses current-format files only.
• If the open WIP PR #13525 (multi-vector) lands, it adds one more element to the same grammar.

What the checks validated

Validated on 2026-11-01 in scratch worktrees of 645d4fe14a20 with the diff applied (removed afterwards), seed DEADBEEF, first for the first version of the diff and again for the final one shown here (format-version paragraph and the DOCS name added after the fact checks):

• Fixed-text check (a validation helper, not the repro): a parser written from the new text, with the corrected value tables transcribed from it, reads the repro's one-field .fnm and a 12-field segment (11 fields plus the parent field setParentField adds) with every element and every new value (FieldBits 0x1, 0x2, 0x8, 0x10; IndexOptions 1, 3 and 5; doc-values 1, 4 and 5; skip index 1; a LongPoint with PointIndexDimensionCount 1 and PointNumBytes 8; FLOAT32 and FLOAT16 vectors) exactly to the footer, every value equal to Lucene's reader: 2 of 2 pass with assertions on and off. It covers the current writer layout (format version 2) only; the sentence on versions 0 and 1 rests on reading read() and FORMAT_CURRENT at the release tags.
• ./gradlew :lucene:core:test --tests TestLucene94FieldInfosFormat --tests TestLucene94FieldInfosFormatFixedJavadoc: 14 tests (the suite's 12, the check's 2), BUILD SUCCESSFUL with assertions on and off, for both versions of the diff.
• :lucene:core:check -x test (ecjLint, forbiddenApis, google-java-format, licenses) and :lucene:core:renderJavadoc: BUILD SUCCESSFUL for both versions; the rendered page of the final one shows the new sequence, the format-version paragraph and 1: indexed as DOCS. google-java-format rewrapped two list items of the first draft and the new paragraph of the final one; the diff shown is the formatted one.
• The minimal repro still fails on the patched tree (expected:<0> but was:<3>, first version), as it must: a javadoc fix leaves the bytes unchanged.
• The full Lucene test suite was not run (javadoc-only change).

Insert IndexOptions and the skip-index byte and add PointIndexDimensionCount and the vector elements to the sequence; leave the tables.

Trade-offs

• Fixes the alignment, which breaks a parser first.
• Leaves the norms-nibble description and the missing codes, so a parser built from the tables still rejects valid files (any SORTED_SET field, any soft-deletes index).

What the checks validated

Not run.

Keep only the prose and say that the .fnm layout is defined by write() and read().

Trade-offs

• Nothing left to drift; fits LUCENE-2946's "high-level" direction.
• Removes the only written description of the default .fnm format, which the file-format overview promises (package-info.java:188).

What the checks validated

Not run.

• First probe (2026-10-28): parsed one default-codec .fnm by the class javadoc and compared every value with Lucene's reader: out of sync at the first field, doc-values codes 4 and 5, FieldBits 0x8 (main, assertions on and off).
• Hunt unit: every file grammar written as data; the documented grammar transcribed only from the javadoc, the layout grammar from the writer; one interpreter over the real bytes of 64 generated default-codec indexes (1114 file checks). For .fnm, 125 of 125 files diverge; the layout grammar parses all of them exactly to the footer (positive control), and 14 of 14 single-element layout mutants across the check's 7 formats are caught (vacuity check), both .fnm ones (drop DocValuesSkipIndex, drop VectorSimilarityFunction) included. An alignment of the two grammars gave the element-level diff: the skip-index byte, PointIndexDimensionCount, the vector elements, FieldBits 0x10, IndexOptions 5 and VectorEncoding 2.
• This is a differential grammar check, not model checking or a proof.