The Lucene90NormsFormat javadoc gives a wrong .nvm Entry grammar: two fields written since 8.0 are missing

Lucene90NormsFormat is the norms format of every default codec since 9.0. Its class javadoc documents the norms metadata file (.nvm) as a typed grammar: each field's Entry is FieldNumber (Int32), DocsWithFieldAddress (Int64), DocsWithFieldLength (Int64), NumDocsWithField (Int32), BytesPerNorm (byte), NormsAddress (Int64). Since the jump tables of LUCENE-8585 (8.0.0, 2019) the writer also stores the jump-table entry count (Int16) and the dense rank power (byte) right after DocsWithFieldLength, in every Entry, and the reader reads them there. A parser written from the javadoc therefore reads NumDocsWithField from the wrong bytes (67108863 instead of 3 for a 3-document segment) and every later value 3 bytes off. The grammar line also labels the file .dvm, the doc-values extension, and the notes call BytesPerNorm "BytesPerValue". It is a documentation defect: Lucene's writer and reader agree, so nothing goes wrong at runtime; the fix is javadoc only.

 *       <p>Norms metadata (.dvm) --&gt; Header,&lt;Entry&gt;<sup>NumFields</sup>,Footer      // :46
 *         <li>Entry --&gt; FieldNumber, DocsWithFieldAddress, DocsWithFieldLength,           // :49
 *             NumDocsWithField, BytesPerNorm, NormsAddress                                 // :50
 *         <li>FieldNumber --&gt; {@link DataOutput#writeInt Int32}                          // :51
 *         <li>DocsWithFieldAddress --&gt; {@link DataOutput#writeLong Int64}
 *         <li>DocsWithFieldLength --&gt; {@link DataOutput#writeLong Int64}
 *         <li>NumDocsWithField --&gt; {@link DataOutput#writeInt Int32}
 *         <li>BytesPerNorm --&gt; {@link DataOutput#writeByte byte}
 *         <li>NormsAddress --&gt; {@link DataOutput#writeLong Int64}                        // :56

The notes below it (:59-67) give the sentinels (FieldNumber -1 ends the metadata; DocsWithFieldAddress -2 when no document and -1 when every document has a norm) and call the bytes-per-norm element BytesPerValue (:61-62, and :76 in the .nvd grammar). The file extension is nvm (METADATA_EXTENSION, :103); .dvm is the doc-values metadata file.

• :100 meta.writeInt(field.number).
• In each of the three branches (no document has a norm :102-106, every document has one :107-111, some do :112-121; the first was only read, no run exercised it): writeLong(docsWithFieldOffset), writeLong(docsWithFieldLength), writeShort(jumpTableEntryCount), writeByte(denseRankPower). In the first two branches both are -1 (:105-106, :110-111); in the sparse branch they are the count IndexedDISI.writeBitSet returns and IndexedDISI.DEFAULT_DENSE_RANK_POWER = 9 (:116-120).
• :123 writeInt(numDocsWithValue), :126 writeByte(numBytesPerValue), :128/:130 the single value or the norms offset. close() writes the -1 end marker and the footer (:70-71).

The reader (Lucene90NormsProducer.readFields, :212-240) reads readLong, readLong, readShort (:223), readByte (:224), readInt (:225), readByte (:226), readLong (:238): the writer's order. Because all three branches write the two values, every Entry is 3 bytes longer than documented.

Reading the documented NumDocsWithField takes the bytes FF FF FF 03, a little-endian 0x03FFFFFF = 67,108,863. Before 9.0 Int32 was big-endian, so on 8.x the same bytes read -256 for a 4-document field (FF FF FF 00).

• Who. Only people who decode .nvm themselves from this javadoc. Lucene's own code never consults it.
• Default codec. Codec.getDefault() is Lucene104 (Codec.java:59), which uses Lucene90NormsFormat (Lucene104Codec.java:76); the codec overview links this class as the norms format (package-info.java:165, :283) and says "Details on each of these are provided in their linked pages" (:188). So the wrong grammar is the documented format of every index written since 9.0.0 that has a field with norms.
• What they get. A wrong NumDocsWithField in the first Entry, then garbage for every later value. The misread count is usually absurd (67,108,863 for a 3-document segment), so it costs debugging time rather than silently corrupting anything.
• What nobody gets. Any runtime effect: writer and reader agree, and the upstream TestLucene90NormsFormat round trip passes.
• Scale and priority. Every release from 8.0.0 (March 2019) to 10.6.0 and main; nobody has reported it in more than seven and a half years, which fits the small audience. Impact minor, priority Low.

Public API only (IndexWriter, Directory.openInput, CodecUtil.checkIndexHeader, IndexInput reads, the default codec), single thread, in the lucene90 codec test package like the upstream norms tests. It needs 9.1.0 or later (the org.apache.lucene.tests test framework); the same file fails unchanged on the 10.6.0 release.

Regression test: optional. A javadoc fix writes the same bytes, so this test keeps failing after the fix (checked); it is evidence for the defect, not a regression test, and Lucene has no test that compares a format javadoc with the bytes. A regression form is this same test with the corrected Entry: insert assertEquals("JumpTableEntryCount", -1, in.readShort()); and assertEquals("DenseRankPower", -1, in.readByte()); before the NumDocsWithField line (validated, see Suggested fix).

cp $REPRO/TestLucene90NormsMetaJavadoc.java $LUCENE/lucene/core/src/test/org/apache/lucene/codecs/lucene90/
cd $LUCENE
./gradlew :lucene:core:test --tests org.apache.lucene.codecs.lucene90.TestLucene90NormsMetaJavadoc -Ptests.seed=DEADBEEF -Ptests.asserts=true
# assertions off: -Ptests.asserts=false (same failure)

# main 645d4fe14a20, ./gradlew :lucene:core:test --tests org.apache.lucene.codecs.lucene90.TestLucene90NormsMetaJavadoc -Ptests.seed=DEADBEEF -Ptests.asserts=true
TestLucene90NormsMetaJavadoc > testDocumentedEntrySequence FAILED
    java.lang.AssertionError: NumDocsWithField expected:<3> but was:<67108863>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:1011BD98D4109FDB]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.apache.lucene.codecs.lucene90.TestLucene90NormsMetaJavadoc.testDocumentedEntrySequence(TestLucene90NormsMetaJavadoc.java:53)
1 test completed, 1 failed
BUILD FAILED

# main 645d4fe14a20, ./gradlew :lucene:core:test --tests org.apache.lucene.codecs.lucene90.TestLucene90NormsMetaJavadoc -Ptests.seed=DEADBEEF -Ptests.asserts=false
TestLucene90NormsMetaJavadoc > testDocumentedEntrySequence FAILED
    java.lang.AssertionError: NumDocsWithField expected:<3> but was:<67108863>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:1011BD98D4109FDB]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.apache.lucene.codecs.lucene90.TestLucene90NormsMetaJavadoc.testDocumentedEntrySequence(TestLucene90NormsMetaJavadoc.java:53)
1 test completed, 1 failed
BUILD FAILED

# 10.6.0 release (4ae946e5a9b9), the same file standalone, -ea, seed DEADBEEF
FAIL tests_run=1 failures=1: java.lang.AssertionError: NumDocsWithField expected:<3> but was:<67108863>

• An exact byte grammar. The class lists the elements in order, gives each a fixed type with a link to the DataOutput method that writes it, and gives sentinel values (Lucene90NormsFormat.java:46-67). Nothing marks it as approximate; package-info.java:188 sends readers to it for the details.
• It used to be exact, and was kept in step with the writer. LUCENE-7475 (9128bdbaf54, 2016) wrote the Lucene70 Entry with five elements; LUCENE-7485 (2fbbcac580f, 2016) added DocsWithFieldLength to the writer and, in the same commit, to the Entry, giving today's text. It matched that writer; the study's probe parses 7.5.0, 7.7.0 and 7.7.3 files exactly with it.
• An oversight in a copy. LUCENE-8585 (c13645bd4c6, 2019) added the two writes and created Lucene80NormsFormat from the Lucene70 javadoc, changing only the title, the codec names and the .nvd DocsWithFieldData link; it also added prose about jump tables to the lucene80 package doc. LUCENE-9705 (4d7b2aebfe0, 2021) copied the text verbatim into Lucene90NormsFormat. No comment on #9631 or on the PR review (apache/lucene-solr#525) mentions the norms grammar.
• Upstream wants the current codec documented well. On that PR a reviewer (jpountz) wrote "Only the current codec needs to have a good description if its file formats". Upstream has fixed this kind of drift as bugs: LUCENE-349 (#1427, docs that "leave out several fields that are read from the file"), LUCENE-713 (#1788), LUCENE-3072 (#4145, "fileformats out of date"). The closest precedent is LUCENE-9353: it moved the terms metadata to a new .tmd file without touching the grammar (#1473, 2020), and about ten months later a separate commit under the same issue revised the terms-dictionary format documentation to match (5f5d1949e92, #90, 2021).

The case for "intended" (rated moderate by reviewers): the codec format classes are @lucene.experimental (Lucene90NormsFormat.java:81); the overview says it "attempts to provide a high-level definition of the Apache Lucene file formats" (package-info.java:54, the LUCENE-2946 policy that moved the docs away from bit-for-bit detail); the supported way to read norms is the codec's own NormsProducer; and the round trip is tested. So nobody should parse .nvm by hand.

Why it fails: this class does not give a high-level description, it gives a typed, ordered grammar, and a precise grammar that is wrong is worse than none. In the LUCENE-2946 thread rmuir said detail is "fine (for now)" for "the simple metadata files", which .nvm is. @lucene.experimental is about API stability, not about whether the docs are true, and the Lucene70 version of this very text was exact. At most the argument makes the fix low priority, which it is.

One fix was validated with gradle in a scratch worktree of 645d4fe14a20; the alternatives were not run.

One file, comments only, +8/-4: add JumpTableEntryCount (Int16) and DenseRankPower (byte) after DocsWithFieldLength in the Entry and the type list; one sentence that they describe the lookup tables of the docs-with-norm set (see IndexedDISI) and are both -1 when no document or every document has a norm; (.dvm) becomes (.nvm); BytesPerValue becomes BytesPerNorm in the notes and the .nvd grammar. The order is the writer's (Lucene90NormsConsumer.java:100-130) and the reader's (Lucene90NormsProducer.java:221-238).

diff --git a/lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90NormsFormat.java b/lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90NormsFormat.java
index b0d60c29f3c..e59757c8245 100644
--- a/lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90NormsFormat.java
+++ b/lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90NormsFormat.java
@@ -43,14 +43,16 @@ import org.apache.lucene.store.DataOutput;
  *       <p>The Norms metadata or .nvm file.
  *       <p>For each norms field, this stores metadata, such as the offset into the Norms data
  *       (.nvd)
- *       <p>Norms metadata (.dvm) --&gt; Header,&lt;Entry&gt;<sup>NumFields</sup>,Footer
+ *       <p>Norms metadata (.nvm) --&gt; Header,&lt;Entry&gt;<sup>NumFields</sup>,Footer
  *       <ul>
  *         <li>Header --&gt; {@link CodecUtil#writeIndexHeader IndexHeader}
  *         <li>Entry --&gt; FieldNumber, DocsWithFieldAddress, DocsWithFieldLength,
- *             NumDocsWithField, BytesPerNorm, NormsAddress
+ *             JumpTableEntryCount, DenseRankPower, NumDocsWithField, BytesPerNorm, NormsAddress
  *         <li>FieldNumber --&gt; {@link DataOutput#writeInt Int32}
  *         <li>DocsWithFieldAddress --&gt; {@link DataOutput#writeLong Int64}
  *         <li>DocsWithFieldLength --&gt; {@link DataOutput#writeLong Int64}
+ *         <li>JumpTableEntryCount --&gt; {@link DataOutput#writeShort Int16}
+ *         <li>DenseRankPower --&gt; {@link DataOutput#writeByte byte}
  *         <li>NumDocsWithField --&gt; {@link DataOutput#writeInt Int32}
  *         <li>BytesPerNorm --&gt; {@link DataOutput#writeByte byte}
  *         <li>NormsAddress --&gt; {@link DataOutput#writeLong Int64}
@@ -58,13 +60,15 @@ import org.apache.lucene.store.DataOutput;
  *       </ul>
  *       <p>FieldNumber of -1 indicates the end of metadata.
  *       <p>NormsAddress is the pointer to the start of the data in the norms data (.nvd), or the
- *       singleton value when BytesPerValue = 0. If BytesPerValue is different from 0 then there are
+ *       singleton value when BytesPerNorm = 0. If BytesPerNorm is different from 0 then there are
  *       NumDocsWithField values to read at that offset.
  *       <p>DocsWithFieldAddress is the pointer to the start of the bit set containing documents
  *       that have a norm in the norms data (.nvd), or -2 if no documents have a norm value, or -1
  *       if all documents have a norm value.
  *       <p>DocsWithFieldLength is the number of bytes used to encode the set of documents that have
  *       a norm.
+ *       <p>JumpTableEntryCount and DenseRankPower describe the lookup tables of that set (see
+ *       {@link IndexedDISI}); both are -1 if no documents or all documents have a norm value.
  *   <li><a id="nvd"></a>
  *       <p>The Norms data or .nvd file.
  *       <p>For each Norms field, this stores the actual per-document data (the heavy-lifting)
@@ -73,7 +77,7 @@ import org.apache.lucene.store.DataOutput;
  *         <li>Header --&gt; {@link CodecUtil#writeIndexHeader IndexHeader}
  *         <li>DocsWithFieldData --&gt; {@link IndexedDISI#writeBitSet Bit set of MaxDoc bits}
  *         <li>NormsData --&gt; {@link DataOutput#writeByte(byte) byte}<sup>NumDocsWithField *
- *             BytesPerValue</sup>
+ *             BytesPerNorm</sup>
  *         <li>Footer --&gt; {@link CodecUtil#writeFooter CodecFooter}
  *       </ul>
  * </ol>

Trade-offs

None for users: no bytes or behaviour change. It does not name the value of DenseRankPower in the sparse case (IndexedDISI.DEFAULT_DENSE_RANK_POWER = 9 today). It leaves two older loose points as they are: the .nvd line uses Data without defining it, and the <Entry>^NumFields line does not show the -1 end marker of the :59 note. The corrected-grammar test checks the dense case only (both values -1); the sparse case was checked by the study's test and the independent reproducer.

What the checks validated

• Stock 645d4fe14a20: the repro fails with assertions on and off. The same test with the corrected Entry (two reads inserted, then read on to the -1 end marker and check the footer position) passes: the fix describes the bytes Lucene already writes.
• With the fix: the repro still fails, as expected (no bytes change). The corrected-grammar test plus TestLucene90NormsFormat and TestLucene90NormsFormatMergeInstance: 63 tests, 6 skipped, 0 failures (in each suite testNCommonBig and testSparseNCommonBig are skipped because the nightly test group is off, and testMergeStability by its assumption on this seed's MockRandom postings format); the corrected-grammar test also passes with assertions off.
• :lucene:core:renderJavadoc, checkGoogleJavaFormat and ecjLintMain: BUILD SUCCESSFUL. The rendered page says "Norms metadata (.nvm)" and never ".dvm" or "BytesPerValue"; its Entry lists all eight elements in the writer's order (from the run's log; the fact check re-rendered the page and saw the same).

lucene/backward-codecs/.../lucene80/Lucene80NormsFormat.java carries the same Entry on the 9.x and 10.x branches (10.6.0 tag: lines 46 and 49-50); main removed it (#15712). Its writer added the same two fields in 8.0.0, so the same text change applies there if the fix is backported.

Trade-offs

Touches a frozen read-only codec on a release branch; worth it only if the fix is backported anyway.

The smallest change that makes the grammar match the bytes; leaves the (.dvm) label and the BytesPerNorm / BytesPerValue mix, which are older (4.9.0 and 7.0.0) and harmless on their own.

Trade-offs

Leaves two known slips in the same ten lines; no reason to split them.

Dropping the two values would change the file format of every existing index, and the reader needs them to open the docs-with-norm set (IndexedDISI).

Trade-offs

A format break for a documentation slip; not a real option.

• Hunt. The grammars are data in a small grammar language; the interpreter compares every parsed value with Lucene's own reader. 64 index configurations (4 fixed and 60 seeded random), 1,114 file checks, assertions on and off.
• Controls. The writer grammar parses every file exactly to the footer (0 failures). Each of 14 single-element mutants of the writer grammar is detected, including dropping the .nvm JumpTableEntryCount or DenseRankPower, so the check is not vacuous.
• Result for .nvm. 112 of 112 files diverge under the javadoc grammar, always first at Entry NumDocsWithField; the same unit found the sibling file-format javadoc defects of .fnm, .si, segments_N, the terms dictionary and the vector formats.
• Minimized to the one-test repro of the Minimal repro section.