Lucene90CompressingTermVectorsFormat accepts a blockSize its writer cannot use, and the first flush with term vectors throws and closes the IndexWriter

Lucene90CompressingTermVectorsFormat's last constructor argument, blockSize, is documented as "the number of chunks to store in an index block", and the constructor rejects only values below 1. Since 8.5.0 the writer uses the value as a DirectMonotonicWriter block shift, the log in base 2 of the chunk count, and DirectMonotonicWriter accepts only 2 to 22. So a custom codec that passes a chunk count such as 1024 (the value the stock format passed up to 8.4.1), or 1 (the constructor's own minimum), builds fine and, in the repro, addDocument returns, but the first flush of a segment with term vectors throws IllegalArgumentException: blockShift must be in [2-22], got 1024 and the IndexWriter closes itself, losing the documents added since the last commit.

The stored-fields twin, Lucene90CompressingStoredFieldsFormat, got exactly the missing range check and a "log in base 2" javadoc in the same change that made the value a shift (LUCENE-9147, 2020); the term vectors format file was not part of that change. Stock codecs pass 10 and are not affected; the committed index stays intact, and nothing returns wrong results.

1. The constructor documents and checks a count. codecs/lucene90/compressing/Lucene90CompressingTermVectorsFormat.java:68: @param blockSize the number of chunks to store in an index block.; :86-88: the only check, if (blockSize < 1) throw new IllegalArgumentException("blockSize must be >= 1"). This is the fix site. The class is @lucene.experimental (:35).
2. The value goes on unchanged, as a shift. vectorsWriter (:100-113) passes it as the last argument of Lucene90CompressingTermVectorsWriter, whose parameter is named blockShift (Lucene90CompressingTermVectorsWriter.java:242), and that hands it to new FieldsIndexWriter(..., blockShift, context) (:281-289).
3. Nothing checks it until the segment is finished. At flush, TermVectorsConsumer.flush calls writer.finish (index/TermVectorsConsumer.java:85), which calls FieldsIndexWriter.finish (Lucene90CompressingTermVectorsWriter.java:725). That writes the shift into the meta file (FieldsIndexWriter.java:116) and calls DirectMonotonicWriter.getInstance(metaOut, dataOut, totalChunks + 1, blockShift) (:125).
4. DirectMonotonicWriter accepts only 2 to 22. util/packed/DirectMonotonicWriter.java:33-34 (MIN_BLOCK_SHIFT = 2, MAX_BLOCK_SHIFT = 22); :46-54 throws IllegalArgumentException("blockShift must be in [2-22], got " + blockShift).
5. Outcome. The exception escapes a segment flush, which IndexWriter treats as aborting: the writer closes (isOpen() == false) and documents added since the last commit are lost; the last commit stays readable, and a new writer with the same codec fails the same way at its next flush with term vectors (all executed). Any flush of such a segment fails, so addDocument itself throws when it triggers an automatic flush (index/DocumentsWriter.java:450, :400-404), and a merge that writes term vectors with this codec reaches the same finish (index/SegmentMerger.java:170-171, :261-267; Lucene90CompressingTermVectorsWriter.java:934). A segment without any term-vector field never creates the term vectors writer: finishDocument returns before initTermVectorsWriter when no field has vectors (index/TermVectorsConsumer.java:119-126, :103-110), and flush skips the finish (:78), so it flushes fine.
6. The twin does it right. Lucene90CompressingStoredFieldsFormat.java:96: @param blockShift the log in base 2 of number of chunks to store in an index block; :117-126 reject anything outside MIN_BLOCK_SHIFT..MAX_BLOCK_SHIFT in the constructor.

Not claimed. Values 2 to 22 are silently read as a shift (asking for "16 chunks" gives 2^16 = 65,536 chunks per block); that is only the documentation mismatch and fails nothing. The exception carries a suppressed CorruptIndexException: checksum passed (...). possibly transient resource issue, or a Lucene or JVM bug for a temp file; it is CodecUtil.checkFooter's generic handling of the earlier exception (FieldsIndexWriter.java:139), not a second defect, though it can mislead.

• Who. Code that constructs Lucene90CompressingTermVectorsFormat itself (it is @lucene.experimental) and passes 1 or 23 and above: for example a chunk count, as the javadoc describes the argument, or 1024, the value the stock format used up to 8.4.1. A custom codec that passed 1024 under 8.4.1 compiles unchanged against 8.5.0 and fails at its first flush there. A GitHub code search found no downstream product that constructs the format, and no public report of it was found since 2020, so the likelihood is rare.
• Default vs opt-in. Opt-in only. Stock code passes 10 (Lucene90TermVectorsFormat.java:170, SortingTermVectorsConsumer.java:40-41).
• Effect. Loud and deterministic, but late and aborting: the error comes at the first flush instead of at construction, the IndexWriter closes, and the documents added since the last commit are lost. No wrong results, no damage to committed data.
• Scale. One document; nothing large is needed.

TestTermVectorsBlockSize (one method, extends LuceneTestCase, package org.apache.lucene.codecs.lucene90.compressing of the lucene/core tests, Apache license header). It is the hunt's minimal repro with one comment shortened to pass the 100-column format check. The test method draws no random values (it uses ByteBuffersDirectory, new IndexWriterConfig() and a FilterCodec over the fixed default codec); LuceneTestCase still randomizes class-level settings such as locale, time zone, similarity and InfoStream from the seed, none of which is on the failing path.

Without a fix it fails through gradle with assertions on and off; with fix A the constructor rejects 1024 and the test returns. The same failure shows through a standalone JUnit runner on the main jars and on the 10.6.0, 10.5.2 and 9.12.3 jars (assertions on and off), which also prints the suppressed CorruptIndexException from FieldsIndexWriter.java:139.

cp $REPRO/TestTermVectorsBlockSize.java $LUCENE/lucene/core/src/test/org/apache/lucene/codecs/lucene90/compressing/
cd $LUCENE && ./gradlew :lucene:core:test --tests org.apache.lucene.codecs.lucene90.compressing.TestTermVectorsBlockSize -Ptests.seed=DEADBEEF   # add -Ptests.asserts=false: same failure

# gradle, scratch worktree at 5adff78bb168, -Ptests.seed=DEADBEEF (assertions on; -Ptests.asserts=false gives the same)
TestTermVectorsBlockSize > testAcceptedBlockSizeCanWriteTermVectors FAILED
    java.lang.IllegalArgumentException: blockShift must be in [2-22], got 1024
        at org.apache.lucene.util.packed.DirectMonotonicWriter.<init>(DirectMonotonicWriter.java:47)
        at org.apache.lucene.util.packed.DirectMonotonicWriter.getInstance(DirectMonotonicWriter.java:158)
        at org.apache.lucene.codecs.lucene90.compressing.FieldsIndexWriter.finish(FieldsIndexWriter.java:125)
        at org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingTermVectorsWriter.finish(Lucene90CompressingTermVectorsWriter.java:725)
        at org.apache.lucene.index.TermVectorsConsumer.flush(TermVectorsConsumer.java:85)
        ...
        at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:586)
        at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:144)
        at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:123)
        at org.apache.lucene.codecs.lucene90.compressing.TestTermVectorsBlockSize.testAcceptedBlockSizeCanWriteTermVectors(TestTermVectorsBlockSize.java:59)
1 test completed, 1 failed

• The constructor's own contract. The javadoc (Lucene90CompressingTermVectorsFormat.java:68) says the value is a chunk count, and the check (:86-88) says every value from 1 up is legal. The writer can use neither 1 nor any value from 23 up.
• The same change fixed the twin, not this file. LUCENE-9147 (136dcbdbbce, Adrien Grand, 2020-02-05; issue #10187, PR apache/lucene-solr#1179) turned the value into a shift: it renamed the writer's parameter blockSize to blockShift, changed the stock argument from 1024 to 10 (Lucene50TermVectorsFormat), and gave CompressingStoredFieldsFormat the MIN_BLOCK_SHIFT..MAX_BLOCK_SHIFT check and the "log in base 2" javadoc. CompressingTermVectorsFormat.java is not in that commit's file list. The class dates from LUCENE-4599 (5837fe83bd8, 2013); its count javadoc and >= 1 check were added by LUCENE-6153 (55eb8b3da39, 2015), when the value really was a count. LUCENE-9705 (f8be421ae1f, 2021) copied them into Lucene90CompressingTermVectorsFormat.
• A second leftover of the old meaning. The same commit switched the test codec's main random range for the value from 1..1024 to MIN_BLOCK_SHIFT..MAX_BLOCK_SHIFT, but CompressingCodec.randomInstance(Random, boolean) still draws it from 1..1024 (lucene/test-framework/.../CompressingCodec.java:97-105; no in-tree caller). It points to an oversight rather than a design; it is not claimed as a bug.
• Tests never reach the gap. The test codec passes one value to both formats (CompressingCodec.java:111-125), so the stored-fields constructor rejects a bad value before the term vectors one sees it. No upstream test passes a term vectors value outside 2 to 22, and no PR review comment, issue comment, MIGRATE or CHANGES entry documents a shift or a range for this argument.

The strongest counter-argument (a summary of the verifiers' intended-behaviour arguments): since 8.5 the value is meant to be a shift; the writer calls it blockShift and every in-tree caller passes a valid shift, so only the javadoc is stale. The class is experimental and the failure is loud.

Why it does not settle it. The code's validation is wrong too, not just the text, and the authors' intent is visible in the twin, which got the range check in the same commit. @lucene.experimental marks API instability, not permission for the check and the writer to disagree. Loud is not harmless here: the failure comes late, closes the writer and drops uncommitted documents, for a value the javadoc describes (1024) or the constructor's own minimum (1). The argument does hold for impact, hence priority Low.

Fix A was built and checked through gradle on 5adff78bb168 in a scratch worktree: the repro with assertions off, then the repro with TestCompressingTermVectorsFormat and TestLucene90TermVectorsFormat, seed DEADBEEF, then :lucene:core:checkGoogleJavaFormat. Option B was not run separately. The hunt's own fix check (the same change) also passed its test and the same two suites, and the verifiers re-ran the fix with their own tests.

In Lucene90CompressingTermVectorsFormat (:68, :86-88) replace the >= 1 check with the twin's MIN_BLOCK_SHIFT..MAX_BLOCK_SHIFT check and message (Lucene90CompressingStoredFieldsFormat.java:117-126), document the value as "the log in base 2 of the number of chunks to store in an index block", and import org.apache.lucene.util.packed.DirectMonotonicWriter. +11/-3 lines.

if (blockSize < DirectMonotonicWriter.MIN_BLOCK_SHIFT
    || blockSize > DirectMonotonicWriter.MAX_BLOCK_SHIFT) {
  throw new IllegalArgumentException(
      "blockSize must be in "
          + DirectMonotonicWriter.MIN_BLOCK_SHIFT
          + "-"
          + DirectMonotonicWriter.MAX_BLOCK_SHIFT
          + ", got "
          + blockSize);
}

Trade-offs

The mistake surfaces at construction, with the twin's message. No behaviour change for any value the writer can use (2 to 22) and no file format change. Code that passes 1 or 23 and above now fails at construction instead of at the first flush.

What the checks validated

Gradle, scratch worktree at 5adff78bb168, seed DEADBEEF: the repro passes with assertions off (the constructor now rejects 1024 and the test returns); the repro plus TestCompressingTermVectorsFormat (22) and TestLucene90TermVectorsFormat (21): 44 tests, 0 failures; :lucene:core:checkGoogleJavaFormat passes with the fix and the test in place.

Also rename blockSize to blockShift in the constructor, the field and the toString label, matching the writer and the twin.

Trade-offs

Clearer, and source compatible (parameter names are not part of the API); the toString output changes.

What the checks validated

Not run separately; the check itself is fix A.

Convert the value to a shift (for example with Integer.numberOfTrailingZeros) so that 1024 works as the javadoc says.

Trade-offs

It would change the meaning of the value 10 that the stock formats pass (Lucene90TermVectorsFormat.java:170, SortingTermVectorsConsumer.java:40-41) and of the valid shifts the test codec draws (CompressingCodec.java:76-83), and a count that is not a power of two has no exact shift.

What the checks validated

Not run.

Method: semi-formal (a proof-obligation audit of a class's contract, witnessed by an executable repro).

• The finder took the constructor's stated contract, its javadoc (Lucene90CompressingTermVectorsFormat.java:68) and its argument check (:86-88), and the obligation it implies: every accepted value must be one the writer can use.
• It followed blockSize through vectorsWriter (:100-113) into Lucene90CompressingTermVectorsWriter (:242, :281-289) and FieldsIndexWriter.finish (:125) to DirectMonotonicWriter's range check (:46-54), which admits only 2 to 22, so the obligation fails for 1 and for every value from 23 up.
• A one-test public-API JUnit (blockSize 1024, one document, a reader or a commit) witnessed it on main with assertions on and off and on the 10.6.0 and 9.12.3 jars; the verifiers, the adjudication and a separate re-run all reproduced it.