The default doc-values format's javadoc says terms are stored in chunks of 16; since 9.0 they are LZ4-compressed blocks of 64

The class javadoc of Lucene90DocValuesFormat, the doc-values format of the default codec, says that terms dictionaries ("Prefix-compressed Binary", used by SORTED and SORTED_SET fields) are "written in chunks of 16, with the first value written completely and other values sharing prefixes" (Lucene90DocValuesFormat.java:101-104 on main 645d4fe14a20). Since 9.0.0 the writer uses blocks of 64 terms: it writes the first term of each block in full, prefix-codes the other 63, and compresses them with LZ4 (Lucene90DocValuesConsumer.addTermsDict, :758-851; TERMS_DICT_BLOCK_LZ4_SHIFT = 6, Lucene90DocValuesFormat.java:217); since 9.2.0 (LUCENE-10536) the first term is the LZ4 dictionary. LUCENE-9843 (a7a02519f0a, 2021) removed the 16-term mode and deleted its constant from this same file, but left the sentence. So only terms 0, 64, 128, ... appear in full in a .dvd file: 17 terms give 1 complete term where the text predicts 2, and 1,000 terms give 16 where it predicts 63. A second, smaller slip: table compression is said to apply for "(< 256)" unique values (:77), but the writer keeps the table up to and including 256 (Lucene90DocValuesConsumer.java:434, size() > 256). Lucene's reader matches its writer, so only the documentation is wrong.

• Lucene90DocValuesFormat.java:101-104 (BINARY section; the SORTED bullet at :110 says terms are "written as Prefix-compressed Binary"): "Prefix-compressed Binary: values are written in chunks of 16, with the first value written completely and other values sharing prefixes. chunk addresses are written as Monotonic-compressed numerics. A reverse lookup index is written from a portion of every 1024th term." No word about LZ4.
• :77: "Table-compressed: when the number of unique values is very small (< 256), and when there are unused "gaps" in the range of values used ... a lookup table is written instead."

• Lucene90DocValuesFormat.java:217-219: TERMS_DICT_BLOCK_LZ4_SHIFT = 6 (64 terms) is the only block size.
• Lucene90DocValuesConsumer.addTermsDict (:758-842, called for SORTED at :755 and SORTED_SET at :1061): when (ord & blockMask) == 0 (:785, every 64th term) it records the block address and writes the term in full, data.writeVInt(term.length) and its bytes (:797-798); every other term goes into a buffer as one token byte (shared-prefix and suffix lengths, with extra vints for long ones) plus its suffix (:803-816). Each block's buffer is written as its length and LZ4.compressWithDictionary(termsDictBuffer, 0, dictLength, ...) with the block's first term as the dictionary (:844-851; the dictionary dates from LUCENE-10536 in 9.2.0, while 9.0.0 and 9.1.0 call LZ4.compress without one). A last block that holds only its first term gets no LZ4 data (:823).
• The reader mirrors it: block mask from TERMS_DICT_BLOCK_LZ4_SHIFT (Lucene90DocValuesProducer.java:1770, :1792) and LZ4.decompress (:1950).
• Table: the writer drops the unique-value table only when uniqueValues.size() > 256 (Lucene90DocValuesConsumer.java:434) and uses it only for 2 or more unique values whose ordinals need fewer bits than the value range (:485-488); the reader rejects only tableSize > 256 (Lucene90DocValuesProducer.java:392-393).

• 17 SORTED terms term00..term16, default codec, one segment: only ord 0 appears in full in .dvd (the javadoc predicts ords 0 and 16).
• The hunt's check: 100, 300 and 1,000 distinct terms give 2, 5 and 16 complete terms (ords 0, 64, 128, ...); chunks of 16 would give 7, 19 and 63.
• 4,096 NUMERIC values with a gap (0 to 254 and 2^40, i.e. 256 distinct): .dvd is 4,169 bytes, 8-bit ordinals per document, a table (the table itself goes to .dvm, Lucene90DocValuesConsumer.java:492-494); with 257 distinct values 24,651 bytes, 48 bits per document.

• Who is hit. People who study the default codec's file formats: to inspect or recover an index, to write a reader in another language, or to start a custom codec from Lucene90DocValuesFormat. The default codec's overview sends them here: lucene104/package-info.java:172 and :288 link the class as the per-document values format, and :188 says "Details on each of these are provided in their linked pages".
• Default vs opt-in. Default: Lucene104Codec uses new Lucene90DocValuesFormat() (Lucene104Codec.java:122), and Lucene90 has only the 64-term LZ4 layout; no setting gives 16-term chunks. A dictionary with one term has no LZ4 data, and the text is right for it.
• What a reader gets wrong. The text puts the block boundaries in the wrong place and does not say that the rest of each block is LZ4-compressed, so it is not enough to read this part of the format. A mental model built from it gets the block size wrong by a factor of 4 and misses the decompression step.
• Scale. None needed: 2 or more distinct terms give an LZ4 block the text does not mention; 17 or more show the wrong block size (ord 16 is not stored in full).
• What it does not do. No wrong result, exception or corruption: Lucene reads what it writes. Impact minor, priority Low; no public report found since 9.0.0 (2021).

• The scan works: ord 0, the first term of the block, is found. Prefix-coded entries hold only a token byte and the suffix ("6" for term16), inside an LZ4-compressed buffer, so a term after the first cannot match. For this input nothing else in .dvd holds a whole term: the reverse terms index writes no term bytes for 17 terms (Lucene90DocValuesConsumer.java:883-893: an entry only at ord 0, with sort key length 0).
• The test sits in the lucene90 test package only for locality; it uses no package-private API. new IndexWriterConfig() with setUseCompoundFile(false) and one flush give one segment with one .dvd file.
• Regression test: none suggested. The fix is javadoc-only; this test asserts the old text, so it keeps failing after the fix by design. The layout itself is already pinned by TestLucene90DocValuesFormat.
• The "(< 256)" slip needs no test: Lucene90DocValuesFormat.java:77 next to Lucene90DocValuesConsumer.java:434 (size() > 256) shows it; the executed sizes are 4,169 bytes for 256 values and 24,651 for 257.

cp TestLucene90DocValuesTermsChunks.java $LUCENE/lucene/core/src/test/org/apache/lucene/codecs/lucene90/
cd $LUCENE && ./gradlew :lucene:core:test --tests org.apache.lucene.codecs.lucene90.TestLucene90DocValuesTermsChunks -Ptests.seed=DEADBEEF -Ptests.asserts=true
# assertions off: -Ptests.asserts=false instead (same failure)

# gradle, main 645d4fe14a20, -Ptests.seed=DEADBEEF -Ptests.asserts=true
TestLucene90DocValuesTermsChunks > testTermsWrittenInChunksOf16 FAILED
    java.lang.AssertionError: expected ords 0 and 16 written completely, actual: [0]
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:7757A8A4DE400BCB]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.assertTrue(Assert.java:42)
        at org.apache.lucene.codecs.lucene90.TestLucene90DocValuesTermsChunks.testTermsWrittenInChunksOf16(TestLucene90DocValuesTermsChunks.java:62)
1 test completed, 1 failed
# assertions off (-Ptests.asserts=false): the same failure, same line

• History. The sentence was written for Lucene45DocValuesFormat (c2f37e0de2d, 2013-08-19, "javadocs/cleanups"), when it was true (16-term address interval), and copied into every later format, into Lucene90DocValuesFormat by LUCENE-9705 (4d7b2aebfe0, 2021-02-22). LUCENE-9663 (8.9.0) added 64-term LZ4 blocks for the non-default BEST_COMPRESSION mode; LUCENE-9843 (a7a02519f0a, 2021-05-06, 9.0.0) removed the modes, and its diff of this file deletes TERMS_DICT_BLOCK_SHIFT = 4 but not the sentence. Its issue (#10882, 18 comments) never mentions the javadoc.
• The real layout is documented elsewhere. CHANGES.txt:4257-4259 (9.2.0, LUCENE-10536): "Doc values terms dictionaries now use the first (uncompressed) term of each block as a dictionary when compressing suffixes of the other 63 terms of the block."
• Tests. TestLucene90DocValuesFormat builds its terms-dictionary cases from TERMS_DICT_BLOCK_LZ4_SIZE (:1116, :1204); LegacyBaseDocValuesFormatTestCase.java:1359 and :1518: "for numbers of values <= 256, all storage layouts are tested".
• The text is meant to be exact. It gives numbers (16, 256, 1024; "blocks of 16384 values" at :89 is right), and the codec overview sends readers to it for the details (lucene104/package-info.java:188). Upstream merged javadoc corrections of this kind (#13801 "Correct Point file extensions in Codec javadocs", #12984 "Improve Javadoc for Lucene90StoredFieldsFormat"), and in 2015 replaced outdated doc-values format javadocs (#7921, LUCENE-6863).

Strongest case for "intended" (built by the cross-model refute and intended reviews, the intended-behaviour verifier and the adjudication): the layout is deliberate (LUCENE-9843, LUCENE-10536); the class is @lucene.experimental (Lucene90DocValuesFormat.java:140); the codec overview calls itself "a high-level definition" (lucene104/package-info.java:54); the bullets are informal prose, not a byte grammar; and "(< 256)" reads like "fits in a byte", a sufficient condition rather than the exact threshold (the bullet also leaves out the bits-saved test).

Why it fails: it defends the code, not the text. "Chunks of 16" is a specific number about the current default format, and Lucene90 has no 16-term path at all. The LZ4 step is not a detail a high-level description may skip, since the bytes after a block's first term cannot be read without decompressing them. @lucene.experimental warns of API changes, not of wrong docs. The constant was deleted from the same file without the sentence, and no discussion decided to keep it. Every verifier and the adjudication rejected the argument for the text; the "(< 256)" half is the weaker one (medium confidence), so it is presented as an imprecision fixed in the same edit.

Three ways to resolve it, all javadoc-only; A was validated. Targets: main and branch_11x, and branch_10_6 if wanted (the two sentences are identical there).

The hunt's text ("blocks of 64 ... the rest of each block is compressed with LZ4, using the first value as a dictionary. Block addresses ..." and "(<= 256)"), re-wrapped by the build's formatter (./gradlew tidy): as written, the hunt's <= 256 line is 101 columns and fails checkGoogleJavaFormat.

diff --git a/lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90DocValuesFormat.java b/lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90DocValuesFormat.java
index 777e76d9792..1c62864fea9 100644
--- a/lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90DocValuesFormat.java
+++ b/lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90DocValuesFormat.java
@@ -74,10 +74,10 @@ import org.apache.lucene.util.packed.DirectWriter;
  * <ul>
  *   <li>Delta-compressed: per-document integers written as deltas from the minimum value,
  *       compressed with bitpacking. For more information, see {@link DirectWriter}.
- *   <li>Table-compressed: when the number of unique values is very small (&lt; 256), and when there
- *       are unused "gaps" in the range of values used (such as {@link SmallFloat}), a lookup table
- *       is written instead. Each per-document entry is instead the ordinal to this table, and those
- *       ordinals are compressed with bitpacking ({@link DirectWriter}).
+ *   <li>Table-compressed: when the number of unique values is very small (&lt;= 256), and when
+ *       there are unused "gaps" in the range of values used (such as {@link SmallFloat}), a lookup
+ *       table is written instead. Each per-document entry is instead the ordinal to this table, and
+ *       those ordinals are compressed with bitpacking ({@link DirectWriter}).
  *   <li>GCD-compressed: when all numbers share a common divisor, such as dates, the greatest common
  *       denominator (GCD) is computed, and quotients are stored using Delta-compressed Numerics.
  *   <li>Monotonic-compressed: when all numbers are monotonically increasing offsets, they are
@@ -98,8 +98,9 @@ import org.apache.lucene.util.packed.DirectWriter;
  *       length}).
  *   <li>Variable-width Binary: one large concatenated byte[] is written, along with end addresses
  *       for each document. The addresses are written as Monotonic-compressed numerics.
- *   <li>Prefix-compressed Binary: values are written in chunks of 16, with the first value written
- *       completely and other values sharing prefixes. chunk addresses are written as
+ *   <li>Prefix-compressed Binary: values are written in blocks of 64, with the first value written
+ *       completely and other values sharing prefixes; the rest of each block is compressed with
+ *       LZ4, using the first value as a dictionary. Block addresses are written as
  *       Monotonic-compressed numerics. A reverse lookup index is written from a portion of every
  *       1024th term.
  * </ul>

Trade-offs

• It states the numbers again (64 and 256), so a future layout change must update the text, as it should have in 9.0; both match the code (TERMS_DICT_BLOCK_LZ4_SIZE, size() > 256).
• "using the first value as a dictionary" is the LUCENE-10536 wording from CHANGES; "Block addresses" replaces the lower-case "chunk addresses".
• The table bullet stays as informal as today: besides at most 256 unique values, the writer needs 2 or more of them and ordinals that take fewer bits than the value range (Lucene90DocValuesConsumer.java:485-488), which "unused gaps" only paraphrases. For the exact condition, a wording such as "a lookup table of up to 256 unique values is written instead when their ordinals need fewer bits than the values themselves" would state it (not validated).
• Like the current text, A stays a high-level description (no token or vint grammar), not a spec to build a parser from.

What the checks validated

• Scratch worktree at 645d4fe14a20, 2026-11-01, JDK 25.0.3. Before the fix the minimal test fails there through gradle with assertions on and off (1 of 1, actual: [0]).
• With the fix, a fix-check test passes 2 of 2 with assertions on and off. It asserts that for 65 terms term00..term64 exactly ords 0 and 64 are in .dvd in full, and that for 4,096 documents .dvd is under 8,192 bytes with 256 distinct values (a table, 8-bit ordinals) and over 8,192 bytes with 257; it printed 4,169 and 24,651 bytes. It does not check the LZ4 bytes, the token format or the dictionary.
• TestLucene90DocValuesFormat, the closest suite: 177 tests, 0 failures (23 nightly tests skipped).
• :lucene:core:check -x test passes on the re-wrapped diff (on the hunt's unwrapped text it failed at checkGoogleJavaFormat, where gradle stopped); :lucene:core:renderJavadoc passes and renders "very small (<= 256)" and "written in blocks of 64, ... compressed with LZ4, using the first value as a dictionary".
• The fix-check test, the suite and the render ran on the same words before the re-wrap, which changes only line breaks inside a comment; the static checks ran on the final diff.
• The minimal test keeps failing after the fix, by design: it asserts the old text.

Replace the number and "chunk addresses", and leave out the LZ4 sentence and the 256 bound.

Trade-offs

Smallest diff, but it keeps the larger omission: the text would still not say that the rest of each block is LZ4-compressed. Not recommended.

What the checks validated

Not run (a subset of A).

Also fix "(< 256)" in backward-codecs Lucene80DocValuesFormat (:78), which repeats it; its "chunks of 16" (:102) is right for its default BEST_SPEED mode (TERMS_DICT_BLOCK_SHIFT = 4, :207), while its BEST_COMPRESSION mode wrote 64-term LZ4 blocks from 8.9.0.

Trade-offs

Frozen formats are rarely edited, and a precise text there must describe both modes.

What the checks validated

Not run.

• The hunt's check (part of a differential file-format check) extracted "written in chunks of 16" and "very small (< 256)" from the javadoc text and checked them on real files: for SORTED fields with 100, 300 and 1,000 distinct terms, which ords appear in clear; for NUMERIC fields of 4,096 documents with 254 to 258 distinct values, the .dvd size. The corrected text matched both checks.
• An independent reproducer wrote its own test (200 terms dvterm-000000 to dvterm-000199, all sharing the 10-byte prefix dvterm-000, plus controls: complete ords are exactly the multiples of 64; 255 values compact, 257 not); on 8.11.4 the same test sees 16-term chunks, so the method detects them where they exist.
• A reviewer from another model family (OpenAI GPT-5.6) ran its own test on 9 seeds (5 with assertions on, 4 off) and a corrected-contract variant (4 of 4 pass); a second review, an intended-behaviour devil's advocate and an adjudication agreed it is real.
• This is a byte-level check of documentation claims, not model checking or a proof.