DOCS_AND_CUSTOM_FREQS: the field length counts terms, but sumTotalTermFreq sums the custom frequencies

Since 10.5.0 a field can be indexed with IndexOptions.DOCS_AND_CUSTOM_FREQS: each term occurs at most once per document and carries an opaque custom value. For such fields IndexingChain adds 1 per term to the field length, so the norm stores a term count (minus overlapping tokens, by default), while the postings keep the custom value, so totalTermFreq and sumTotalTermFreq sum the custom values. FieldInfo#isTermDocField says the frequency "for statistical and scoring purposes is treated as 1" and FieldStats says sumTotalTermFreq "is also the sum of each document's field length"; neither holds on this option, and nothing documents the exception.

The measured consequence: BM25 and every SimilarityBase model take avgdl from sumTotalTermFreq / docCount and compare it with dl from the norm, so on such fields dl counts terms while avgdl is inflated by the mean custom value per posting (without overlapping tokens). For two documents with custom frequency 5 everywhere, dl = 2 and 4 meet avgdl = 15 instead of 3, and the short document's lead for query a drops from 10.2% to 2.2%. In the recorded default-BM25 scope probes, hit sets, deletions, CheckIndex and pruned top-k matched the controls, and nothing throws at normal scale. Both halves are deliberate and tested upstream, so this is a documentation and API-contract defect, with the BM25 effect as an open question: the #15584 author asked about BM25 on such fields, and nobody discussed dl against avgdl.

IndexingChain.java:2053-2064, once per token:

if (fieldInfo.isTermDocField()) {
  invertState.length = Math.addExact(invertState.length, 1);                       // :2054-2055
} else {
  invertState.length =
      Math.addExact(invertState.length, invertState.termFreqAttribute.getTermFrequency()); // :2057-2059
}

1. Length. On DOCS_AND_CUSTOM_FREQS the length grows by 1 per term; on every other option by the custom value. Similarity#computeNorm encodes it (minus overlaps by default) into the norm (Similarity.java:151-161). This is the #15584 change, made so that large custom values no longer overflow the per-document length ("too many tokens", the #11086 / LUCENE-10048 use case).
2. Postings and term statistics. The posting freq is still the custom value (FreqProxTermsWriterPerField.java:125, :177, via getTermFreq() at :208-209); a second occurrence of a term in one document is rejected (:193-194). The postings writer sums freq() into totalTermFreq (PushPostingsWriterBase.java:139-140), as the TermsEnum and Terms javadocs define it and CheckIndex enforces.
3. Search-time statistics. IndexSearcher#fieldStats sums Terms#getSumTotalTermFreq over the leaves (IndexSearcher.java:1144-1159); termStats passes totalTermFreq through (:1130-1133).
4. Consumers. BM25Similarity#avgFieldLength is sumTotalTermFreq / docCount (BM25Similarity.java:143-145); SimilarityBase sets the field's token count and average length from sumTotalTermFreq (SimilarityBase.java:85-86). The Similarity class javadoc recommends exactly that (Similarity.java:54-55).

So only on this option is dl in terms while avgdl is in custom-value units: without overlapping tokens, avgdl is multiplied by sumTotalTermFreq / sumDocFreq, the mean custom value per posting (30 / 6 = 5 in the example). On legacy DOCS_AND_FREQS custom frequencies both sides sum the custom values; on DOCS fields both count unique terms.

sumDocFreq is 6 and sumTotalTermFreq 30 on both stock fields. The probe is a printing test, not the repro.

• Who. Applications that index a field with IndexOptions.DOCS_AND_CUSTOM_FREQS (released in 10.5.0; the IndexOptions enum is @lucene.experimental).
  • Code that reads FieldInfo#isTermDocField, FieldStats or the Similarity javadoc to decide what the statistics mean, for example a custom similarity or a distributed-statistics layer, gets a wrong contract.
  • Scores change with norms on (a TextField-derived type keeps them) and a length-normalising similarity: the default BM25Similarity or any SimilarityBase model (DFR, IB, LM). Without overlapping tokens, dl / avgdl shrinks by the mean custom value per posting: 5 in the example, where the short document's lead drops from 10.2% to 2.2%. Custom values may go up to Integer.MAX_VALUE (upstream's testNoOverflowInt), and larger values can weaken length normalisation much further; how much depends on b, k1 and how the values are spread. One document with large values raises avgdl for the whole field.
• Not affected. Fields without norms (every document then has dl = 1), BM25 with b = 0, similarities that ignore length (upstream's tests score this option with pass-through similarities, and BaseSimilarityTestCase runs it with norms omitted), legacy DOCS_AND_FREQS custom frequencies, and, in the recorded default-BM25 scope probes, everything except scoring: on an index with deletions, 120 queries gave the same hit sets as a legacy custom-frequency index and CheckIndex was clean on both; on 150 queries pruned top-1/top-10 equalled exhaustive scoring; scores stayed finite. The probes cover the tested cases, not every query, codec or similarity.
• Silent. At normal scale, no exception and no log; only explain() shows the mismatch between dl and avgdl.
• No scale needed. One document shows it. Nothing wrong is written to the index: IndexSearcher combines the statistics at search time, so a search-time fix (option B) changes the statistics it gives the scorers without a reindex; the raw postings statistics (TermsEnum, Terms) stay in custom-value units.
• How likely: rare. The option is new (April 2026) and meant for opaque per-term scores with custom scoring; code search found no downstream user beyond an Elasticsearch back-compat test.

Deterministic, single thread, no random() in the test body; the test framework's per-seed codec choice does not change the values (seeds DEADBEEF, 5EED1234 and C0FFEE, assertions on and off).

The custom value comes from a test-framework Token (setTermFrequency(5), i.e. the TermFrequencyAttribute), as upstream's TestCustomTermFreq feeds this option with its own CannedTermFreqs stream; the javadocs of the documented feeders (TermFrequencyAttribute, DelimitedTermFrequencyTokenFilter) still say DOCS_AND_FREQS (option A fixes them).

It asserts the documented identity only and does not say which half upstream should change; both are deliberate and tested. It passes with option B and keeps failing with the javadoc-only option A, so it is a regression test only for a behavioural fix. The BM25 consequence is the probe in the full explanation above, deliberately not part of the repro.

On 10.x, call new IndexSearcher(r).collectionStatistics("f") instead of fieldStats("f"): main renamed CollectionStatistics to FieldStats in fa0e704d2a0 (#15929), and 10.x carries the same javadoc sentence.

cp $REPRO/TestTermDocFieldStatsVsFieldLength.java $LUCENE/lucene/core/src/test/org/apache/lucene/search/
cd $LUCENE && ./gradlew :lucene:core:test --tests TestTermDocFieldStatsVsFieldLength -Ptests.seed=DEADBEEF -Ptests.asserts=true
# assertions off fails the same way: -Ptests.asserts=false

TestTermDocFieldStatsVsFieldLength > testSumTotalTermFreqIsSumOfFieldLengths FAILED
    java.lang.AssertionError: sumTotalTermFreq expected:<1> but was:<5>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:916F8584BC265B64]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.apache.lucene.search.TestTermDocFieldStatsVsFieldLength.testSumTotalTermFreqIsSumOfFieldLengths(TestTermDocFieldStatsVsFieldLength.java:60)

1 test completed, 1 failed

# the same test on IndexOptions.DOCS_AND_FREQS passes: length 5 == sumTotalTermFreq 5

• The javadoc says the opposite of what the index does. FieldInfo.java:717-720: "Returns true if the field has custom term frequencies indexed that are to be treated as opaque values, while the term frequency for statistical and scoring purposes is treated as 1." Upstream's tests contradict both halves: TestCustomTermFreq#testTotalTermFreqTermDoc (:314-344) expects totalTermFreq 92 = 42 + 50 and 188 = 128 + 60, and TestSimilarity#testTermDocSimilarity (:207-248) expects a TermQuery score of 17, the custom value, under a similarity whose tf(freq) is freq (:48).
• FieldStats documents an identity that held until 10.5.0. FieldStats.java:55-58: sumTotalTermFreq "is the sum of TermStats#totalTermFreq() across all terms. It is also the sum of each document's field length across all documents." The sentence dates from LUCENE-8007 (2017); the same repro on DOCS_AND_FREQS passes on 9.12.3, 10.4.0, 10.5.0, 10.5.1, 10.5.2, 10.6.0 and main. Similarity.java:54-55 tells similarity authors to compute the average length from it.
• Nothing documents the exception. No javadoc, MIGRATE.md or CHANGES text mentions termdoc statistics; the CHANGES entry covers the length half and ends mid-sentence: "IndexWriter counts their terms rather than summing their freqs. Use" (lucene/CHANGES.txt:919-920). The TermFrequencyAttribute and DelimitedTermFrequencyTokenFilter javadocs still say the field "must be indexed with IndexOptions#DOCS_AND_FREQS" (2017), although upstream's tests feed custom frequencies into DOCS_AND_CUSTOM_FREQS fields.
• The question was asked; the units were never discussed. On #15584 (2026-01-17) msokolov wrote "I guess someone might try using BM25 simlarity with this field, or some scoring that assumes that totalTermFrequency > termFrequency(doc)?", and rmuir answered "Searching and pruning would be my concerns. Look at BaseSimilarityTestCase for the former." The PR then ran BaseSimilarityTestCase on the option with norms omitted and synthetic statistics, and added a pass-through-similarity score test, so nothing exercises dl against avgdl. On #11086 (2026-01-16) msokolov asked what would happen "if someone were to, say, compute the default Similarity over such a field (we don't plan to do that, but someone could, and it shouldn't break in a horrible way)"; nobody replied. No comment on either thread mentions the length / sumTotalTermFreq split.
• Precedent for one unit. LUCENE-8025 (#9073) and LUCENE-8031 (#9079), the same kind of mismatch on DOCS fields, were fixed as bugs in 2017 and 2018; on one test collection rmuir measured BM25 MAP 0.0531 before and 0.1390 after. LUCENE-7854 (#8905) asked for custom frequencies to give lengths "same as if i stacked".

The strongest case for "intended" (the intended-behaviour check, the study's cross-model lenses and the adjudication rated it moderate; the 2026-10-31 refute re-run and the cross-model fact check rated it strong for the code behaviour, the fact check "weak for the documentation as written"): each half is deliberate and tested. testFieldInvertState pins the length at 2 for the freqs {42, 128} (TestCustomTermFreq.java:612-631) and testTotalTermFreqTermDoc pins totalTermFreq at 92. Terms and TermsEnum define the totals as sums of freq(), and CheckIndex enforces it. The custom values are "scores", not counts; the author called BM25 on such fields a non-goal that "shouldn't break in a horrible way"; on the first, attribute-based draft of #15584 rmuir wrote that it "won't fix the issue of all kinds of crazy/undefined behavior at search-time. It is just allowing it to happen at index-time."; and rmuir called avgdl "just a pivot" on LUCENE-8000 (#9048, open). BM25Similarity#avgFieldLength is protected and documented as sumTotalTermFreq / docCount, so an application can override it. And the FieldStats sentence is already loose: on a DOCS field "a a" the length is 2 while sumTotalTermFreq is 1, and with a stacked synonym BM25 sees dl 1.0 against avgdl 2.0.

Why it fails: it defeats a claim that the statistics code is wrong, which this report does not make. It does not defeat the documentation defect: the isTermDocField javadoc is false for statistics and for scoring whichever side upstream keeps, and nothing documents the split. On DOCS fields BM25 still compares dl and avgdl in one unit, and synonyms follow the documented discountOverlaps rule; only the termdoc option fails both readings (length vs sumTotalTermFreq, dl vs avgdl). "Not horrible" is why the impact is minor, not why it is correct.

Both options were built and checked with gradle in a scratch worktree of 645d4fe14a20 (OpenJDK 25.0.3, 2026-10-31). Which is right is a design choice for upstream; option A is needed under every choice, because the "scoring purposes" half of the FieldInfo javadoc is false whatever happens to the statistics.

Javadoc only, +16/-6 lines in 6 files: FieldInfo#isTermDocField (below); IndexOptions.DOCS_AND_CUSTOM_FREQS adds "The field length counts each term once, while TermsEnum#totalTermFreq() and Terms#getSumTotalTermFreq() sum the custom freqs, so a similarity that compares a document's length with the average length (such as BM25) sees the two in different units."; FieldStats (sumTotalTermFreq) and the Similarity class javadoc name the exception; TermFrequencyAttribute and DelimitedTermFrequencyTokenFilter allow DOCS_AND_CUSTOM_FREQS next to DOCS_AND_FREQS.

/**
 * Returns true if the field is indexed with {@link IndexOptions#DOCS_AND_CUSTOM_FREQS}: each term
 * occurs at most once per document and carries a custom term frequency, an opaque value rather
 * than a count. Only the field length ({@link FieldInvertState#getLength()}) counts each term
 * once; {@link PostingsEnum#freq()} returns the custom value, and {@link
 * TermsEnum#totalTermFreq()} and {@link Terms#getSumTotalTermFreq()} sum the custom values.
 */
public boolean isTermDocField() {

Trade-offs

• No behaviour change, so it fits a bugfix release.
• BM25's units stay split; the javadoc then says so instead of promising the opposite.
• The minimal repro keeps failing, so it is not a regression test for this option.
• The study's audit, written from a committer's view, would merge exactly this: a javadoc fix "saying that the statistics and scoring see the custom value, and that length-normalising similarities such as BM25 compare a term-count dl with an avgdl in custom-value units".

What the checks validated

checkGoogleJavaFormat (after applyGoogleJavaFormat rewrapped one line), renderJavadoc and ecjLintMain pass for lucene:core and lucene:analysis:common.

+22/-1 lines in IndexSearcher.java: for a field whose FieldInfo#isTermDocField() is true, termStats reports totalTermFreq = docFreq and fieldStats reports sumTotalTermFreq = sumDocFreq, the term count that the field length uses. The raw index statistics are unchanged.

public TermStats termStats(Term term, int docFreq, long totalTermFreq) throws IOException {
  if (isTermDocField(term.field())) {
    // custom freqs are opaque values, not counts: each term counts once per document
    totalTermFreq = docFreq;
  }
  // This constructor will throw an exception if docFreq <= 0.
  return new TermStats(term.bytes(), docFreq, totalTermFreq);
}
// fieldStats(): sumTotalTermFreq += termDoc ? terms.getSumDocFreq() : terms.getSumTotalTermFreq();
// isTermDocField(field): the FieldInfo of the first leaf that has the field

Trade-offs

• Makes the TermStats and FieldStats that IndexSearcher supplies count-based for termdoc fields, so "for statistical purposes treated as 1" and both FieldStats identities hold for those statistics, and puts dl and avgdl in one unit: the example then scores exactly like the legacy DOCS_AND_FREQS field (0.1545 and 0.1402). The public TermsEnum#totalTermFreq() and Terms#getSumTotalTermFreq() stay in custom-value units; CheckIndex and testTotalTermFreqTermDoc are untouched; no reindex.
• Behaviour change: BM25 and SimilarityBase scores on existing termdoc fields with norms change (LM, DFR and IB models also see count-based collection statistics), so it may suit main better than a 10.x bugfix release.
• Both methods must change together: changing only fieldStats would break the documented totalTermFreq <= sumTotalTermFreq and trip the assertion in SimilarityBase.java:80.
• Subclasses that override termStats/fieldStats (distributed statistics) keep the raw values, and the "scoring purposes" half of the javadoc still needs option A's wording.
• isTermDocField(field) takes the FieldInfo of the first leaf that has the field. Within one index a field's index options cannot change (FieldInfo#verifySameIndexOptions), but an IndexSearcher over a MultiReader that combines a termdoc leaf and a legacy leaf of the same field name would apply the first leaf's rule to every leaf; a real patch should decide per leaf or reject the mix.
• One FieldInfos lookup over the leaves per call (it stops at the first leaf with the field); not benchmarked.

What the checks validated

• Stock: the minimal test fails. With the patch it passes, with assertions on and off.
• :lucene:core:test --tests "org.apache.lucene.search.*" --tests TestCustomTermFreq (216 classes, including TestSimilarity, search.similarities.* and the repro): 2,355 tests, 0 failures with seed DEADBEEF and assertions on, and again with seed C0FFEE and assertions off.
• checkGoogleJavaFormat, ecjLintMain and forbiddenApisMain for core pass.
• The BM25 probe with the patch gives avgdl 3.0 and the legacy field's scores.

The LUCENE-8025 pattern: BM25 and SimilarityBase use sumDocFreq / docCount as the average length on termdoc fields.

Trade-offs

FieldStats carries no index options, so it needs a new flag; it fixes BM25 but not the javadocs, and the repro keeps failing.

Revert the IndexingChain.java:2054-2055 branch.

Trade-offs

Restores one unit but brings back the per-document "too many tokens" overflow that #15584 removed.

Method: non-formal (a randomized differential and metamorphic harness, the hunt's tier for executed testing without a model checker or prover).

• A hunt unit tests index-options invariance: family W indexes the same weighted terms as natural repetitions, as legacy custom frequencies on DOCS_AND_FREQS, and as DOCS_AND_CUSTOM_FREQS, and relation R2 expects bit-identical scores within a score class.
• R2 had to put DOCS_AND_CUSTOM_FREQS with norms in a class of its own, because its field length counts terms; with norms off R2 held bit-exactly over 440 seeds. Checking that class split against the FieldStats javadoc exposed the inconsistency.
• The finder minimised it into a 4-test public-API JUnit on the stock jars of main a9779ef3c193; the registry later reduced it to the one-test repro above.