MoreLikeThis keeps its lowest-scoring selected terms when its query hits the boolean clause limit

MoreLikeThis builds a query from the most interesting terms of a text: it scores each candidate by tf*idf and keeps the best maxQueryTerms in a priority queue. A BooleanQuery may hold at most IndexSearcher.getMaxClauseCount() clauses (1024 by default), so when more terms are selected, createQuery catches TooManyClauses and returns the clauses it has. It fills the query by popping the queue, and since 4.10.0 the queue pops the lowest score first, so the clauses that fit are the weakest terms and the best ones are dropped. With 1100 selected terms, 76 of them scoring twice the others, the 1024-clause query holds none of the 76; with maxQueryTerms 1024 it holds all of them. It happens through MoreLikeThis.like(...), MoreLikeThisQuery and the XML query parser's LikeThisQuery, with assertions on and off, and only when more terms are selected than the clause limit (the defaults, 25 terms and 1024 clauses, never reach it). LUCENE-5795 (2014) flipped the queue order for a faster top-N selection and kept the old loop, which up to 4.9.1 kept the best terms. Skipping the lowest-scoring terms that cannot fit before building the query (+11 lines) fixes it, and the whole queries module passes with that change.

• MoreLikeThis (lucene-queries, org.apache.lucene.queries.mlt) finds documents similar to a source: a like-text, a document id or a field map. It scores every candidate term of the source by tf * idf and keeps the best maxQueryTerms (default 25) as SHOULD clauses of a BooleanQuery.
• The clause limit: a BooleanQuery.Builder refuses more than IndexSearcher.getMaxClauseCount() clauses (1024 by default) and throws IndexSearcher.TooManyClauses.

1. createQueue keeps the top min(maxQueryTerms, #eligible terms) terms (eligible: passing the minTermFreq, minDocFreq and maxDocFreq filters) in a bounded priority queue whose top is the lowest score (MoreLikeThis.java:624-677, comparator at :628-629). The selection is right.
2. createQuery pops that queue, so it sees the terms lowest score first, and adds a SHOULD TermQuery per term (:593-616).
3. After 1024 clauses the builder throws (BooleanQuery.java:82-84), and createQuery does catch (IndexSearcher.TooManyClauses _) { break; } (:611-613). Every term still in the queue, the highest-scoring ones, is dropped.
4. So whenever more terms are selected than the clause limit, the query loses the best #selected - limit terms. No exception, no warning.

Up to 4.9.1 the queue popped the best term first (FreqQ.lessThan returned fa > fb), so the same break kept the best terms. LUCENE-5795 (4.10.0) made the queue lowest-first to bound it cheaply and kept the loop.

Default clause limit 1024; an index where all 1100 words have the same idf; a like-text in which 76 words occur twice and 1024 once (the diagram probe ran 45 sampled values from 25 to 2000; every row matches them, and the ranges follow from the mechanism):

Lowered limit: with IndexSearcher.setMaxClauseCount(3), the default maxQueryTerms 25 and words a to f occurring 6, 5, 4, 3, 2 and 1 times, the query is {d, e, f}; with maxQueryTerms 3 it is {a, b, c}.

• Who. Applications that let MoreLikeThis select more terms than the global clause limit: maxQueryTerms above 1024 on a source with more than 1024 eligible terms, through MoreLikeThis.setMaxQueryTerms, MoreLikeThisQuery.setMaxQueryTerms (rewritten by IndexSearcher) or the XML query parser's LikeThisQuery, whose maxQueryTerms attribute comes from the query text; or a global limit lowered below the selection size (IndexSearcher.setMaxClauseCount is public).
• Not by default. The defaults are 25 terms (MoreLikeThis), 5 (MoreLikeThisQuery) and 20 (XML LikeThisQuery). The other in-tree users, the classification module's k-nearest-neighbour classifiers and Luke, never set maxQueryTerms, so they are reached only if the global limit is below 25.
• Eligible terms. By default a term must occur at least twice in the source and in at least 5 documents, and a field without term vectors is parsed up to 5,000 tokens, so the trigger needs long sources, several fields (the queue counts terms over all fields) or relaxed filters.
• Effect. Every such call returns a query built from the least interesting selected terms; the best #selected - 1024 terms are lost. The result is non-monotonic: asking for more terms makes the query worse. The query can give worse "more like this" relevance (no relevance benchmark was run): other terms change which documents match as well as how they rank, and MoreLikeThisQuery also sets minimumNumberShouldMatch from the clause count. Nothing fails, nothing is corrupted, and the query never exceeds the clause limit.
• How likely. Rare: large maxQueryTerms values are unusual, and no report was found since 4.10.0 (2014). Anyone configured that way is hit on every long source without any sign, so the absence of reports is weak evidence.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<[alpha, beta]> but was:<[delta, gamma]>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:120)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.queries.mlt.TestMoreLikeThisClauseLimit.testLoweredClauseLimitKeepsBestTerms(TestMoreLikeThisClauseLimit.java:63)

• Purpose. The class javadoc: "Since you're trying to maximize a tf*idf score, you're probably most interested in terms with a high tf" (MoreLikeThis.java:63-64), and "return a set of representative terms" (:79).
• Best-first wording. The private retrieveTerms(Reader, String) returns "the most interesting words in the document ordered by score, with the highest scoring, or best entry, first" (:875-876), the order the createQuery loop was written for; stale since LUCENE-5795. retrieveInterestingTerms: "we just want to return the top words" (:895, :920).
• The limit was anticipated. DEFAULT_MAX_QUERY_TERMS: "Return a Query with no more than this many terms." with @see IndexSearcher#getMaxClauseCount (:225-232); createQuery catches TooManyClauses and returns the query it has (:609-615).
• History. From the first check-in (2005) to 4.9.1 the queue popped best-first; LUCENE-5795 is listed under 4.10.0 "Optimizations": "MoreLikeThisQuery now only collects the top N terms instead of collecting all terms from the like text when building the query." (lucene/CHANGES.txt:11932-11934). Its 9 comments discuss the bounded queue, its struct, the added test and the commits; none mentions the pop order or the clause limit. Swapping only MoreLikeThis.java flips the result both ways on 4.9.1 and 4.10.0.
• Core precedent. TopTermsScoringBooleanQueryRewrite "only uses the top scoring terms so it will not overflow the boolean max clause count" and caps its size at getMaxClauseCount() (MultiTermQuery.java:146-169).
• Tests. No lucene/queries test reaches TooManyClauses in MoreLikeThis; testTopN uses 25 terms under 1024. testSmallSampleFromCorpus asserts the lowest-first clause order and testCustomFrequency relies on the lowest-score boost normalisation; the recommended fix keeps both, and both pass with it.

The strongest works-as-designed case (the independent reproducer, the devil's advocate, the cross-model reviews and the adjudication): setMaxQueryTerms promises only a maximum ("the maximum number of query terms that will be included in any generated query"), and DEFAULT_MAX_QUERY_TERMS points at getMaxClauseCount, which could mean "keep maxQueryTerms at or below the clause limit". The catch could be just a heap backstop: BooleanQuery.Builder.add says its check exists "to prevent run-away 'rewriting' of bad queries from creating BQ objects that might eat up all the Heap". @jpountz wrote in #8931 (LUCENE-7880, a thread asking to allow more clauses per query): "I think of the max clause count as something that should never be changed. What is the use-case for having more than 1024 clauses?" That question arguably applies to the main trigger itself (maxQueryTerms above 1024). And upstream tests encode the ascending clause order.

Why it fails: no javadoc says values above the limit are unsupported, the setter accepts them, and the code chooses to return a truncated query. Up to 4.9.1 the same code kept the best terms, and the change came with an optimisation whose discussion never covered it. Asking for 1100 terms instead of 1024 removes all 76 best ones, which is hard to defend for a class that exists to choose the most interesting terms. The #8931 remark is about changing the limit at all, in a thread about raising it; this bug needs neither: the main trigger leaves the limit at its default, the query stays at 1024 clauses, and createQuery deliberately truncates instead of rejecting, so the only question is which 1024 terms survive. Clause order does not change matching, and the fix keeps it anyway. Verdict: not intended; a regression from LUCENE-5795. The expectation comes from the class's purpose, the pre-4.10 behaviour and the best-first javadoc, not from an explicit promise about the clause-limit path.

Two fixes were built and run in a scratch worktree at a9779ef3c193. A changes only how the query is built and is recommended; B limits the selection itself.

Before the build loop, pop and discard q.size() - getMaxClauseCount() terms, which are the lowest-scoring ones, still seeding the boost normaliser from the lowest selected score. +11 lines in MoreLikeThis.java; the finder's change (+10) with a reworded, 3-line comment.

float bestScore = -1;

// q pops the lowest score first. If more terms were selected than the clause limit allows,
// skip the lowest-scoring ones so that the best terms are kept (clause order and boost
// normalisation do not change).
int skip = q.size() - IndexSearcher.getMaxClauseCount();
while (skip-- > 0) {
  scoreTerm = q.pop();
  if (boost && bestScore == -1) {
    bestScore = scoreTerm.score;
  }
}

while ((scoreTerm = q.pop()) != null) {

Trade-offs

• Only the truncating case changes: a query that fits is built exactly as today. Clauses stay in ascending order, and with setBoost(true) a kept term gets the boost it would get in the untruncated query.
• retrieveInterestingTerms and createQueue are untouched; the catch (TooManyClauses) stays as a backstop, no longer reached unless another thread lowers the global clause limit while the query is built.
• Cost: on the truncating path only. The skip loop pops #selected - limit terms; stock already pops one term that does not fit, so the net extra is #selected - limit - 1 pop() calls (O(log n) each). Not benchmarked.
• It edits the same bestScore lines as the fix for the sibling boost-normalisation bug (a selected score of 0 gives NaN boosts); if both are filed together, the skip loop needs the same positive-score rule.

What the checks validated

Gradle in a scratch worktree at a9779ef3c193, seed DEADBEEF: the whole :lucene:queries:test with the 4 repro classes added, 55 classes, 520 tests, 0 failures (the module's own skips: 5 with assertions on, 6 off); all 8 repro tests that fail on stock pass, and the randomized oracle (which also covers setBoost(true)) has 0 wrong of 600 checks. :lucene:queries:check -x test with the fix and the primary test (google-java-format, ecj lint, forbidden APIs): BUILD SUCCESSFUL. On ea57360912ed the same change also made the MoreLikeThisQuery and XML LikeThisQuery probe pass.

limit = min(maxQueryTerms, #terms, IndexSearcher.getMaxClauseCount()) in createQueue (+6/-2 lines). maxQueryTerms above the limit then behaves exactly like maxQueryTerms equal to the limit, and the queue never holds entries that cannot be used.

Trade-offs

retrieveInterestingTerms(...) also goes through createQueue, so it would return at most getMaxClauseCount() terms although it builds no query; keeping it unchanged needs the clamp on the like(...) paths only. With setBoost(true) the boosts of a truncated query are normalised by the lowest kept score instead of the lowest selected one. When several terms tie at the cut, fix A and fix B can keep different ones of them (the comparator has no tie-breaker); both keep a top-scoring set.

What the checks validated

Gradle with assertions on, the whole :lucene:queries:test with the 4 repro classes: 55 classes, 520 tests, 0 failures. The independent fact check also ran it with assertions off on the 4 repro classes and TestMoreLikeThis: 19 tests, 0 failures, 1 skipped. Not run through check.

Also keeps the best terms.

Trade-offs

Changes the clause order that testSmallSampleFromCorpus asserts and, unless adjusted, the boost normaliser. Not run.

A javadoc note on setMaxQueryTerms and MoreLikeThisQuery.setMaxQueryTerms.

Trade-offs

A legitimate maintainer choice, but the silent, non-monotonic truncation stays for anyone who misses the note, including XML queries that set the attribute. Not run.

Method: bounded random translation validation against a reference semantics (the hunt's "semi-formal" tier).

• The hunt plan noted from code reading that createQuery adds clauses in the pop order of the score queue and stops at the first TooManyClauses, and its oracle asked whether MoreLikeThis keeps a lower-scoring term while dropping a higher-scoring one, "incl. at the clause limit". So the lead came from reading; the comparison confirmed it by execution.
• The finder wrote a reference model of MoreLikeThis's term selection from its javadoc and compared it with the real queries jar on random inputs over the like(int), like(Map) and like(String, Reader) entry points: check R3, "the kept terms are the top-scoring eligible terms", failed 851 times in 118,800 checks (seed DEADBEEF), all at the clause limit and none elsewhere.
• A bounded-exhaustive model of createQueue/createQuery (61,208 cases) confirmed the condition: a failure exactly when the clause limit is below min(maxQueryTerms, #terms) and a dropped term outscores a kept one (18,480 cases); a best-first mutant fails none.
• The finder then wrote the public-API JUnit test, and independent checkers reproduced it.