IndriDirichletSimilarity(2000f) scores differently from IndriDirichletSimilarity(): the mu constructor silently uses another collection model

IndriDirichletSimilarity implements "Bayesian smoothing using Dirichlet priors as implemented in the Indri Search engine": a document's score for a term is log((tf + mu * p(w|C)) / (|D| + mu)), where the collection probability p(w|C) comes from a CollectionModel. The class ships its own IndriCollectionModel, cf / |C|, and the no-argument constructor, documented as "the default mu value of 2000", uses it. The one-argument constructor IndriDirichletSimilarity(float mu) only assigns mu, so Java inserts super(), and LMSimilarity() installs DefaultCollectionModel, (cf + 1) / (|C| + 1). So new IndriDirichletSimilarity(2000f) and new IndriDirichletSimilarity() have the same mu and print the same LM IndriDirichlet(2000.000000), but score differently: on a 4-document index of 25 tokens, "president" (cf 3) gets p(w|C) = 4/26 instead of 3/25, and doc 0 scores -1.871553 instead of -2.119101. Any positive mu is affected (at mu = 0 the model drops out of the formula): every score and explain() is shifted, and close documents can swap places; on large indexes the shift is a fraction of a percent. Nothing throws and nothing is written to the index. Every release since 8.9.0, the first with the class, is affected.

• Dirichlet smoothing (Indri): a document's score for a term is log((tf + mu * p(w|C)) / (|D| + mu)), where tf is the term's count in the document, |D| the document length, mu the smoothing strength and p(w|C) the collection probability: how common the term is in the whole field.
• Collection model: the strategy object of LMSimilarity that computes p(w|C). Two exist here: IndriCollectionModel, cf / |C| (Indri's estimate: occurrences of the term divided by all tokens of the field), and LMSimilarity.DefaultCollectionModel, (cf + 1) / (|C| + 1).

1. IndriDirichletSimilarity has five constructors (IndriDirichletSimilarity.java:40-66). The three that take a CollectionModel pass it on. IndriDirichletSimilarity() is this(new IndriCollectionModel(), 2000) (:64-66). IndriDirichletSimilarity(float mu) is { this.mu = mu; } (:54-56): no this(...) or super(...), so the compiler inserts super().
2. LMSimilarity() is this(new DefaultCollectionModel()), "Creates a new instance with the default collection language model" (LMSimilarity.java:55-58). DefaultCollectionModel returns (totalTermFreq + 1) / (numberOfFieldTokens + 1) (:150-152); IndriCollectionModel returns totalTermFreq / numberOfFieldTokens (IndriDirichletSimilarity.java:111-113).
3. LMSimilarity.fillBasicStats stores the model's value once per term (LMSimilarity.java:68-73) and score() uses it for every document (IndriDirichletSimilarity.java:69-72); explain() prints it as "collection probability" (LMSimilarity.java:75-80).
4. Both models return getName() = null and getName() of the similarity prints only mu (IndriDirichletSimilarity.java:96-99), so both constructors give toString() = LM IndriDirichlet(2000.000000).

(cf + 1)/(|C| + 1) - cf/|C| = (|C| - cf) / (|C| (|C| + 1)), about 1/|C|. So the numerator tf + mu * p(w|C) grows by about mu / |C|: from 241 to 308.7 in the 25-token example (mu 2000), but only by 0.0015 at 1.3 million tokens. The relative change of p(w|C) is about 1/cf, so rare terms move most when the shift matters at all. The shift vanishes at mu = 0, where the term mu * p(w|C) is 0 under both models (one of the upstream test's cases), and when cf = |C|, where both models give 1.

Three related javadoc slips: IndriCollectionModel's javadoc says "divided by the total number of tokens + 1" but the code adds nothing (IndriDirichletSimilarity.java:101-104); LMSimilarity's class javadoc describes DefaultCollectionModel without its +1 (LMSimilarity.java:33-35); and DefaultCollectionModel's own javadoc names only the +1 in the denominator (LMSimilarity.java:140-143).

• Who is hit. Users of IndriDirichletSimilarity(float mu), the natural way to tune mu, installed with IndexSearcher#setSimilarity. Any positive mu, assertions on and off. The upstream TestIndriDirichletSimilarity itself only uses this constructor.
• Effect. For mu > 0, scores and explain() differ from Indri's cf / |C| estimate, which the class description implies, and from IndriDirichletSimilarity() with the same mu (except in the degenerate case cf = |C|). The set of matching documents does not change.
• How likely a ranking change is. A credibility audit ran 1,984 random 2- and 3-term BooleanQuery queries over 5 synthetic corpora (1,458 to 1,309,450 tokens, mu 2000): no top hit changed, no top-10 set changed, one top-10 order changed. The largest relative score change was 0.019% at 1.3 million tokens and 0.5% to 6.8% on the small corpora. The audit sorted its top-10 lists from Weight#scorer scores, not from IndexSearcher#search, and synthetic corpora say nothing about real workloads. The ranking flips the checkers built are on tiny constructed corpora: in 26 of 146 swept 4-document indexes (130 to 3,030 tokens) two close documents, ranks 3 and 4 of 4, swapped places (scores read through Weight#scorer, the count printed, not asserted); in one 3-document index of 35 tokens the top hit of IndexSearcher#search changed. They show that flips can happen, not how often.
• Where it shows on each release. Since 10.3.0 a plain top-k TermQuery search under this similarity returns no hits at all, a separate bug (LFV-190, negative scores dropped by BatchScoreBulkScorer). So on 10.3.0 to 10.5.2 and main the wrong values show in explain(), Scorer-level use and multi-clause BooleanQuery search; on 8.9.0 to 10.2.x also in plain search (on 10.2.2, search(TermQuery("president"), 10) returns doc 0 at -2.119101 with the no-argument constructor and -1.871553 with (2000f)). This bug survives a fix of LFV-190 and of open PR #463.
• Not reached. The case where the shift would be large, the smoothing score of an absent term in IndriAndQuery, is not reached for TermQuery clauses on main (the audit, by reading).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<-0.6931472> but was:<-0.4057149>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:577)
       org.junit.Assert.assertEquals(Assert.java:701)
       org.apache.lucene.search.similarities.TestIndriDirichletMuConstructor.testExplicitDefaultMuScoresLikeDefaultConstructor(TestIndriDirichletMuConstructor.java:50)

• Contract (implied, not word for word). Class javadoc: "Bayesian smoothing using Dirichlet priors as implemented in the Indri Search engine" (IndriDirichletSimilarity.java:23-25); CHANGES for LUCENE-9537: "the IndriDirichletSimilarity from the academic Indri search engine" (lucene/CHANGES.txt:5346-5347). The class javadoc writes the formula with an unspecified P(t|C) (:27-30), so cf / |C| is implied, not stated. Indri computes cf / |C| for a term that occurs in the collection (a checker read it in the Indri source, and the fact check re-read it), which is what IndriCollectionModel does; for a term that never occurs Indri uses 1 / (2|C|), which Lucene's TermStats never allows, and Indri's own default mu is 2500. The no-argument constructor is documented only as "the default mu value of 2000" (:63).
• The sibling keeps one model. LMDirichletSimilarity, whose constructor signatures this class mirrors, has LMDirichletSimilarity() { this(2000); } (LMDirichletSimilarity.java:71-73). Only IndriDirichletSimilarity switched its no-argument constructor to another model.
• No intent for the difference. The PR that added the class (apache/lucene-solr#2097, 10 conversation and 28 inline review comments) and LUCENE-9537 (#10577, 20 comments) never mention the collection model or the constructors. #13845 added the discountOverlaps constructor and discussed only that flag, not Indri. None of them says that choosing a mu should also switch the collection model.
• The upstream test does not pin it. TestIndriDirichletSimilarity builds only new IndriDirichletSimilarity(mu) with a random mu (:48), asserts generic properties, no values, and passes with the fix.
• The +1's original reason does not apply. LUCENE-3430 (2011) added the +1 in the numerator because SpanQueries sometimes scored terms with ttf = 0 (the +1 in the denominator dates from the first LMSimilarity, LUCENE-2959). On main TermStats and FieldStats reject zero counts (TermStats.java:77-80, FieldStats.java:102-104), and the no-argument constructor already uses cf / |C| on every input. That removes the zero-count reason; it does not rule out that a maintainer sees the +1 as deliberate smoothing.
• Before the port (a checker's reading of third-party code): in the class author's own projects, Lucindri (release 1.1, 2020) and IndriLuceneImpl, the same constructor bodies sat on a base class whose default model is cf / |C|, so both constructors gave Indri's estimate. Rebasing the class on LMSimilarity changed the mu constructor's meaning silently. Nobody upstream has said so.

The strongest case for "works as written" (built by the devil's advocate; the cross-model reviews rated it moderate): by Java's rules the mu-only constructor inherits LMSimilarity's documented default model, its javadoc promises only mu, the constructor signatures mirror LMDirichletSimilarity's, the (cf + 1)/(|C| + 1) default is a deliberate Lucene choice, LMSimilarity is @lucene.experimental (IndriDirichletSimilarity itself carries no marker), and the upstream test has always run the mu constructor.

Why it fails: it explains the mechanism but not why new IndriDirichletSimilarity() and new IndriDirichletSimilarity(2000f) should differ. The LMDirichletSimilarity precedent points the other way (all its model-less constructors share one model, and it claims no Indri fidelity), the +1's reason does not apply on main, the test asserts no values, and the author's original code gave cf / |C| on both paths. Verdict: not intended, but the contract is implied; a maintainer may answer with a javadoc note or another API change (deprecating the mu-only overload, naming the model in toString()) because a code fix changes scores for current users; the fix below is one reasonable resolution, not the only one.

One option was built and validated with gradle; a documentation-only alternative was not run.

IndriDirichletSimilarity(float mu) becomes this(new IndriCollectionModel(), mu), like the no-argument constructor. In the same change, IndriCollectionModel's javadoc drops the "+ 1" it does not compute, and LMSimilarity's class javadoc states DefaultCollectionModel's +1. A third slip is not in the validated diff: DefaultCollectionModel's own javadoc (LMSimilarity.java:140-143) leaves out its numerator +1, a one-line javadoc edit you may add. Suggested CHANGES entry: "IndriDirichletSimilarity(float mu) now uses IndriCollectionModel, like the no-argument constructor. It used LMSimilarity's DefaultCollectionModel, (cf+1)/(|C|+1), so it scored differently from IndriDirichletSimilarity() with the same mu."

  /** Instantiates the similarity with the provided μ parameter. */
  public IndriDirichletSimilarity(float mu) {
-    this.mu = mu;
+    this(new IndriCollectionModel(), mu);
  }
 ...
   * Models {@code p(w|C)} as the number of occurrences of the term in the collection, divided by
-   * the total number of tokens {@code + 1}.
+   * the total number of tokens.

Trade-offs

Scores of current IndriDirichletSimilarity(float) users with mu > 0 change, toward Indri's formula and toward IndriDirichletSimilarity(). Indexes need no change: the model is applied at search time. No material cost change is expected (one model object per similarity, the same arithmetic per score); not benchmarked. A maintainer may want it on main only. Open PR #463 edits score() a few lines away, so the two may conflict textually.

What the checks validated

• Scratch worktree at a9779ef3c193, gradle, JDK 25: the new test 3 of 3 with assertions on and off; TestIndriDirichletSimilarity 2 of 2 and TestIndriAndQuery 2 of 2; 140 of 140 over 20 iterations of the three.
• Every class in org.apache.lucene.search.similarities plus TestIndri*: 28 classes, 123 tests, 0 failures. :lucene:core:check -x test (google-java-format, ecjLint, forbiddenApis) and :lucene:core:renderJavadoc pass.
• Standalone runner with the patched class shadowed: the checker's ranking test passes (both constructors rank doc1 first); the independent reproducer's claim tests pass and only its diagnostic, which asserts the old behaviour, flips.
• Not run: the whole core suite (on main only TestIndriDirichletSimilarity calls the mu constructor).

Keep the code. Say in the mu constructor's javadoc that it uses LMSimilarity.DefaultCollectionModel, (cf+1)/(|C|+1), and that new IndriDirichletSimilarity(new IndriCollectionModel(), mu) gives Indri's estimate; fix the three javadoc slips. Optionally give IndriCollectionModel a non-null getName() so toString() tells the two apart.

Trade-offs

No score change for anyone, but new IndriDirichletSimilarity() and new IndriDirichletSimilarity(2000f) still differ, and without the naming change they still print the same name. The naming change alters toString() output.

Both constructors would then use DefaultCollectionModel, as in LMDirichletSimilarity.

Trade-offs

Removes the inconsistency but abandons the class's documented Indri estimate and changes scores for users of the default constructor (by reading; not run).

Method: bounded-exhaustive translation validation of scoring components against their documented formulas (hunt unit "scoring components vs documented formulas", harness part B1, 2026-10-27, main a9779ef3c193).

• The harness indexed every document of 1 to 4 tokens over a 3-letter alphabet (120 documents, including a 4-byte and a 3-byte letter), scored each TermQuery hit through Weight#scorer, and compared it with log((tf + mu * cf/|C|) / (|D| + mu)).
• IndriDirichletSimilarity() matched for unboosted queries (its remaining mismatches are boosted queries, whose boost score() ignores, which open PR #463 also changes). IndriDirichletSimilarity(3f) mismatched in all 1,080 of its formula checks.
• Reading the code traced that to the constructor. The finder then wrote the public-API JUnit, checked it on 10.5.2 and 9.12.3, and validated the one-line fix with the upstream Indri tests.
• Tier: semi-formal (a bounded-exhaustive check of the real jar against exact semantics); it proves nothing beyond its bounds.