Lucene's single-term (InvertableType.BINARY) indexing path counts each value twice in the field length, so untokenized DOCS_AND_FREQS fields with norms store a doubled length in their norms and get lower BM25 scores

Since 9.6.0 an untokenized value can be indexed through a single-term path that skips the TokenStream (InvertableType.BINARY, IndexingChain.invertTerm). StringField and KeywordField use it, and so does IndexWriter#addBatch for untokenized BinaryColumn and DictionaryColumn columns. The path updates the field length twice for every value (invertState.length++; and invertState.length = Math.addExact(invertState.length, 1);), although FieldInvertState#getLength is the "total number of terms in this field" and a BINARY value is "a single value" with term frequency one. For an untokenized field with DOCS_AND_FREQS and norms, Similarity#computeNorm uses that length, so the norm encodes a length of 2k instead of k for k values and BM25 scores the document lower than the identical document indexed through a TokenStream (0.058813408 vs 0.082873434 in the repro). Postings, matching and CheckIndex are unaffected; StringField and KeywordField are DOCS-only with norms omitted. Both increments came in with the path itself in #12139; deleting the plain invertState.length++; fixes it.

• Field length and norm: while a field is inverted, FieldInvertState counts its terms (getLength(), "total number of terms in this field"). At the end of the document Similarity#computeNorm turns that count into the norm, which BM25 reads back as the document length dl.
• Two inversion paths: IndexableField#invertableType() chooses how a field is inverted. TOKEN_STREAM (a plain Field) runs the analyzer or a one-token stream; BINARY indexes binaryValue() as one term without a TokenStream. StringField and KeywordField return BINARY, and addBatch uses it for untokenized BinaryColumn and DictionaryColumn columns.

1. PerField.invert calls invertTerm for BINARY and invertTokenStream for TOKEN_STREAM (IndexingChain.java:1950-1968 on main a9779ef3c193). The addBatch row pass (processRowColumns, :847) wraps a BinaryColumn or DictionaryColumn in a BinaryColumnAdapter (ColumnFieldAdapter.java:51-62) whose invertableType() is BINARY for indexed untokenized columns (:244-250), and reaches the same invert through invertAndStore (IndexingChain.java:1477-1487).
2. invertTerm (IndexingChain.java:2127-2170) rejects tokenized fields and anything with positions or term-vector positions, offsets or payloads (:2136-2144), then runs invertState.position++; (:2147), invertState.length++; (:2148), termsHashPerField.start(field, first); (:2149) and invertState.length = Math.addExact(invertState.length, 1); (:2150). It adds the term once (:2152), so the postings get freq 1, but the length grows by 2.
3. invertTokenStream adds 1 per token (or the custom term frequency) in a single Math.addExact (IndexingChain.java:2053-2064).
4. PerField.finish stores similarity.computeNorm(invertState) (IndexingChain.java:1926-1944). Similarity#computeNorm uses getLength() for every index option except DOCS, which uses the unique-term count (Similarity.java:151-161). The length is encoded with SmallFloat.intToByte4 (Similarity.java:160), so a DOCS_AND_FREQS field gets the norm of length 2k instead of k for k values; a DOCS field is unaffected under the stock similarities. The norm byte equals the length up to 40 and is lossy above (SmallFloat.java:139-165): one value gives norm 2 instead of 1, 21 values the code 41 (decoded length 42) instead of 21.
5. BM25's average length is sumTotalTermFreq / docCount (BM25Similarity.java:143-146), which is right; only each document's own length is doubled, so every BINARY document's dl/avgdl is twice its correct value (2 instead of 1 in the repro, where each document's true length equals the average).

One index, default IndexWriterConfig (BM25), field type untokenized DOCS_AND_FREQS with norms. One value per document, doc 0 through a plain Field and doc 1 as BINARY: norms [1, 2], TermQuery scores [0.082873434, 0.058813408], explain shows dl=1.0 and dl=2.0 with avgdl=1.0. Two values per document: norms [2, 4], the same scores. addBatch, doc 0 with addDocument and docs 1 and 2 as a BinaryColumn and a DictionaryColumn: norms [1, 2, 2], scores [0.06069608, 0.043074638, 0.043074638]. The same with IndexOptions.DOCS: norms [1, 1], equal scores. With values a, b, a in one document the BINARY path writes norm 6 while sumTotalTermFreq is 3; the TokenStream path and MemoryIndex give 3.

• Who. Applications with an untokenized indexed field whose FieldType has DOCS_AND_FREQS and norms enabled (omitNorms defaults to false, FieldType.java:41) that goes through the BINARY path: addDocument with a custom IndexableField or Field subclass whose invertableType() returns BINARY (public API, 9.6.0 on; IndexableField#invertableType() has no default, so every implementation that does not extend Field picks one), or the @lucene.experimental IndexWriter#addBatch with an indexed untokenized BinaryColumn or DictionaryColumn (10.5.0 on), where no override is needed.
• What goes wrong. Norms and BM25 scores only, silently: no exception, CheckIndex clean. Matching, deletions and FieldExistsQuery are unchanged (executed through addDocument: 40 random documents, 2 deleted, once with a plain Field and once with a BINARY field, assertions on and off; addBatch reaches the same invertTerm by code reading).
• Size of the effect. BM25's term-frequency factor is tf = freq / (freq + k1 * (1 - b + b * dl / avgdl)) (BM25Similarity.java:219, :272-273, :307-328), so doubling dl/avgdl lowers each score by a fraction that depends on the document's frequency and length: 29% for freq 1 at dl = avgdl, as in the repro. The scores do not drop uniformly, and ranking can flip even inside one field where every document takes the BINARY path (executed by the fact check, public API, seed DEADBEEF, main with assertions on and off and 9.6.0): docs A = [x], B = [x, x, y] and a third with 8 distinct values (avgdl 4), query tag:x. All through a plain Field: A 0.3081991 < B 0.3159688; all as BINARY: A 0.26857352 > B 0.25753623 (A loses 12.9%, B 18.5%), so the order flips; with the fix the BINARY scores equal the plain-Field ones. The doubling also shifts the field's weight against other fields and against documents of the same field indexed through a TokenStream, for example addDocument with a plain Field next to addBatch columns.
• Default vs opt-in. Default IndexWriterConfig, codec and similarity; the opt-in part is the field type, which no stock Lucene field uses.
• Not affected. StringField and KeywordField (DOCS, norms omitted); DOCS-only fields with norms under the stock computeNorm, which counts unique terms. For DOCS fields getLength() is doubled too (executed: the independent reproducer's recording BM25Similarity and the diagram probe print getLength() [1, 2] with norms [1, 1]), so a custom Similarity that reads it there would see the doubling (read).
• Scale. Deterministic for every value that takes the path, independent of the index size, up to the overflow limit below.
• Side effect (read, not executed). The Math.addExact overflow check fires after about 2^30 values in one field of one document instead of about 2^31: today the 2^30th value makes Math.addExact (IndexingChain.java:2150) throw a raw ArithmeticException; with the fix the limit moves to the 2^31st value, still a raw ArithmeticException, while the TokenStream path wraps the same overflow in IllegalArgumentException (:2061-2064).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<1> but was:<2>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:647)
       org.junit.Assert.assertEquals(Assert.java:633)
       org.apache.lucene.index.TestBinaryInvertableTypeNorm.testSingleValueNorm(TestBinaryInvertableTypeNorm.java:49)

• The contract. FieldInvertState.java:124-131: getLength(): "Get total number of terms in this field." InvertableType.java:25-32: BINARY: "The field should be treated as a single value whose binary content is returned by IndexableField#binaryValue(). The term frequency is assumed to be one. [...] If the same value is provided multiple times, the term frequency will be equal to the number of times that this value occurred in the same document."
• Every sibling counts 1 per value (executed): the same field type through the TokenStream path, and MemoryIndex's single-term path added in the same commit (storeTerm, info.numTokens++ once per value, MemoryIndex.java:968-969). The same writer's sumTotalTermFreq (1 per value) disagrees with its own norm (2 per value).
• Lucene's own test accepts the trigger. TestDocumentWriter#testIndexBinaryValueWithoutTokenStream (TestDocumentWriter.java:435), added in #12139, indexes a BINARY field with six legal field types, including untokenized DOCS_AND_FREQS with setOmitNorms(false) (:502-509). It checks postings and term vectors, never norms (:543-592). The upstream column tests set omitNorms(true) on their untokenized types.
• History. invertState.length++; is the form the TokenStream path used before LUCENE-7854 (2017) replaced it with the Math.addExact form; invertTerm has both, already in the first commit of #12139, unchanged since. The PR is about attribute overhead; its 12 comments discuss API shape and naming and never mention length, norms or scores. CHANGES lists it under 9.6.0 as "GITHUB#12139: Faster indexing of string fields." No javadoc, CHANGES or MIGRATE entry restricts norms on BINARY fields.

The strongest works-as-designed case, built by the intended-behaviour verifier, a review by a model from another family (OpenAI GPT-5.6) and the adjudication: BINARY is a performance fast path whose only stock users omit norms, so norms on a BINARY field are an unsupported combination; addBatch is @lucene.experimental and its javadoc does not promise the same index as addDocument; and the addDocument trigger needs a custom invertableType() override.

Why it fails: #12139 promised only speed and no document restricts norms on this path; the PR's own test lists the trigger field type as legal; overriding invertableType() is public API that StringField and KeywordField use themselves; two adjacent increments of the same counter serve no performance or semantic purpose; and @lucene.experimental covers API stability, not wrong index data, while the released trigger is addDocument. For DOCS fields under a custom Similarity that reads getLength() (not the stock case) the argument is narrow reach, not correctness: getLength() still promises the total number of terms and reports twice that. Every verifier and the cross-model reviews rated the intended argument weak; the cross-model fact check of this doc rated it weak for addDocument and moderate for the experimental addBatch path and the custom-Similarity DOCS case, and found no plausible purpose for the duplicate increment.

One fix was applied in a scratch worktree at a9779ef3c193 and run through gradle (seed DEADBEEF), together with a norms check added to the existing BINARY test; the alternatives were not run.

One line in IndexingChain.invertTerm. Each value then adds 1 to the field length, like the TokenStream path and MemoryIndex, so norms and BM25 scores match the TokenStream path for the same field type; postings and term vectors are unchanged. A second diff adds 8 lines to TestDocumentWriter#testIndexBinaryValueWithoutTokenStream: for each legal field type with norms, norm 2 for the two DOCS_AND_FREQS values and 1 for DOCS.

       invertState.setAttributeSource(null);
       invertState.position++;
-      invertState.length++;
       termsHashPerField.start(field, first);
       invertState.length = Math.addExact(invertState.length, 1);

Trade-offs

• Existing segments keep the norms they were written with until the documents are reindexed, because merges copy the stored norms; old and new segments of such a field score differently in the meantime. Worth a CHANGES line.
• None for new documents: the result equals the TokenStream path.

What the checks validated

• Stock main a9779ef3c193: the two repro files fail 3 of 4 with assertions on and off; TestDocumentWriter with the added norms check fails 1 of 15 ("expected:<2> but was:<4>").
• With the fix: the repro files pass 4 of 4 with assertions on and with assertions off; TestDocumentWriter with the norms check passes 15 of 15.
• With the fix, the closest suites pass: in core TestDocumentWriter, TestIndexableField, TestFieldReuse, TestFieldInvertState, TestNorms, TestCustomNorms, TestOmitNorms, TestCustomTermFreq, TestDocInverterPerFieldErrorInfo, TestColumnBatchIndexing, the six document.column classes, TestKeywordField and TestField, plus the two repro classes (20 classes, 258 tests, 0 failures); all of lucene/memory (3 classes, 51 tests).
• ecjLint and checkGoogleJavaFormat pass for core with the repro tests and the norms check in place; both diffs apply cleanly to a clean tree.
• Earlier, in the hunt (fix shadowed onto the stock jars): the finder's, the independent reproducer's, the GPT reviewers' and the adjudication's tests pass, as do upstream TestDocumentWriter (15 run, 0 failed) and the column-batch tests the reviewers ran; the same deletion makes the introducing commit pass.
• Not run: the full lucene/core suite; no benchmark (the change removes one increment).

Also gives 1 per value.

Trade-offs

Drops the overflow check that the TokenStream path has. Not recommended.

Optional hardening on top of the recommended fix: throw IllegalArgumentException("too many tokens for field ...") as invertTokenStream does (IndexingChain.java:2061-2064).

Trade-offs

Only matters at about 2^31 values in one field of one document (2^30 before the fix); today invertTerm throws a raw ArithmeticException there, and this makes both paths fail the same way.

Method: bounded-exhaustive translation validation on the stock jar (semi-formal tier), in a hunt unit on the column-batch addBatch API.

• The harness indexes every one of 26 single-column specs (Long, Binary, Dictionary, Vector and TokenStream columns with different field types) through addBatch and, as the reference, the same documents row by row through addDocument, then compares the two indexes: dense and every sparse pattern for 0 to 3 documents, a document before and/or after the batch, and 4 IndexWriterConfigs, with assertions on and off (29,408 scenarios).
• It reported 1,242 norm mismatches, all from the one DictionaryColumn spec that is untokenized with DOCS_AND_FREQS and norms: batch norm 2, row norm 1. With the one-line fix shadowed onto the jar the mismatches went to 0.
• The root cause was then located by reading invertTerm, which showed that the path is shared with addDocument; the finder added the addDocument trigger (a Field subclass returning BINARY), which reaches back to 9.6.0.
• Found 2026-10-25 on main ea57360912ed; then checked by an independent reproducer, an intended-behaviour devil's advocate, two executed reviews by a model from another family (OpenAI GPT-5.6), an adjudication, a history and prior-report follow-up and a credibility audit.