NearestFuzzyQuery and KNearestFuzzyClassifier throw "boost must be a positive float" when most of the index is deleted documents not yet merged away

NearestFuzzyQuery, the query behind KNearestFuzzyClassifier in the classification module, weights each term with idf = ln((numDocs + 1) / (docFreq + 1)) + 1, taking numDocs from reader.numDocs() (live documents only) and docFreq from reader.docFreq(), which still counts deleted documents that are not merged away yet. When the deleted copies push docFreq + 1 above e × (numDocs + 1), the idf is negative, and new BoostQuery(termQuery, negativeScore) in rewrite() throws IllegalArgumentException: boost must be a positive float (since 8.0.0, when LUCENE-7996 made BoostQuery reject negative boosts; 7.x silently returned negative scores). The smallest case with a live document is 5 documents with "apple", 4 deleted: numDocs 1, docFreq 5, idf −0.0986 (with retained soft deletes and no live document, 2 documents already throw). It needs more than about 63% of all documents deleted but still present: reached with the stock Log*MergePolicy before a merge, soft-delete retention (even after forceMerge(1)) and NoMergePolicy, not with the default TieredMergePolicy in any tested shape. Using reader.maxDoc() keeps idf ≥ 1; the highlighter had the same mismatch: LUCENE-1355 clamped docFreq, then LUCENE-4289 switched to maxDoc().

1. KNearestFuzzyClassifier.assignClass and getClasses (KNearestFuzzyClassifier.java:105-133) call knnSearch (:135-148), which adds one NearestFuzzyQuery for the input text (addTerms once per text field) as a MUST clause, next to a MUST WildcardQuery on the class field and the optional filter query, and calls indexSearcher.search(bq.build(), k) (:147). Those clauses do not help: rewrite takes its statistics from the whole reader.
2. NearestFuzzyQuery.rewrite (NearestFuzzyQuery.java:217-264) collects the terms on the searcher's whole reader (:219, :223). The private addTerms reads int corpusNumDocs = reader.numDocs(); (:140), then per analyzed term int df = reader.docFreq(startTerm); (:155); when the term itself is absent, df is the average docFreq of its variants (:173-178), which counts deleted documents too.
3. Each variant's score is (editBoost * editBoost) * idf(df, corpusNumDocs) (:187), with idf = (float) (Math.log((docCount + 1) / (double) (docFreq + 1)) + 1.0) (:198-200). The squared factor is never negative (it can be 0 for an edit of a one-character term, and 0 × a negative idf is −0.0, which BoostQuery rejects too), so a variant's boost is rejected exactly when the idf is negative: docFreq + 1 > e × (numDocs + 1).
4. rewrite wraps each variant as new BoostQuery(tq, st.score) (:248, :256), and BoostQuery's constructor throws IllegalArgumentException("boost must be a positive float, got " + boost) for a negative boost (BoostQuery.java:38-44). It is a plain check, not an assertion.
5. numDocs() counts live documents (IndexReader.java:292-298), docFreq also counts deleted ones (IndexReader.java:401-408), and docFreq ≤ maxDoc(). So with maxDoc() at :140 the idf is always ≥ 1.

• The minimal repro: 5 documents with "apple", 4 deleted under NoMergePolicy, an NRT reader: numDocs 1, maxDoc 5, docFreq 5; count(NearestFuzzyQuery("apple")) throws boost must be a positive float, got -0.098612286 = ln(2/6) + 1, with -ea and -da, on main and on the 10.6.0 jars.
• Sweep, 1 live document plus d deleted copies: d = 0, 1, 2, 3 give idf 1.0, 0.595, 0.307, 0.084 and 1 hit; d = 4 to 8 throw exactly the predicted idf: -0.0986, -0.253, -0.386, -0.504, -0.609.
• The same failure, executed on main and 10.6.0: KNearestFuzzyClassifier.assignClass("apple"); the misspelled input "appel" (-0.0631 = 0.8² × -0.0986, through the variant average); a commit-point reader instead of an NRT reader; one document soft-updated 4 times while SoftDeletesRetentionMergePolicy keeps the old versions, then forceMerge(1); LogDocMergePolicy at its defaults.
• Not affected: no deletions; 3 of 4 deleted; the default IndexWriterConfig, which had already dropped the deletes when the reader opened (maxDoc 1).
• Threshold in general: since docFreq ≤ maxDoc, it needs maxDoc + 1 > e × (numDocs + 1), more than about 1 − 1/e = 63% of the reader's documents deleted but present (80% with 1 live document). Then an input term whose df (its docFreq, or for an absent term its variants' average, deleted copies included) exceeds e × (numDocs + 1) − 1 makes rewrite throw if one of its variants is kept. For long input that is necessary, not sufficient: the query keeps only the 300 highest-scoring variants overall (MAX_NUM_TERMS, :58), so enough positive-scored variants can push the negative ones out; the one-term repro always keeps its variant. Inputs without such a term still work.

• Who is hit. Applications that use KNearestFuzzyClassifier (@lucene.experimental) or NearestFuzzyQuery (public, not tagged experimental) from the opt-in classification module, on an index or reader where more than about 63% of the documents are deleted but still present. A request throws when a kept fuzzy variant comes from a term above the threshold (in long input, positive-scored variants can push it out of the 300 kept); others work.
• Default vs opt-in. The default IndexWriterConfig (TieredMergePolicy) dropped the deletes in every test, including a probe with up to 1,000 added-then-deleted documents; its deletesPctAllowed, "the maximum percentage of doc id space taken by deleted docs", is 20% by default and at most 50% (TieredMergePolicy.java:120-124), both under the 63% needed, but it is a target for merge selection, not a guarantee, and heavy churn on large segments was not tested. The state does arise with supported setups: the stock LogByteSizeMergePolicy or LogDocMergePolicy before a merge; SoftDeletesRetentionMergePolicy (itself @lucene.experimental), whose "main purpose ... is to implement retention policies for document modification to vanish in the index" (SoftDeletesRetentionMergePolicy.java:40-44; forceMerge(1) keeps the retained versions); NoMergePolicy; or a reader view that hides documents through live docs (no deletions at all; a custom wrapper, an informational run).
• Scale. No scale needed: 5 documents. The failure is immediate and loud; nothing wrong is returned and nothing is written. It lasts as long as the index state does.
• Likelihood: rare. No downstream caller of these classes was found: Solr's classification processor uses KNearestNeighborDocumentClassifier and SimpleNaiveBayesDocumentClassifier, and code search of Solr, Elasticsearch and OpenSearch finds neither class. No public report was found in the documented searches, although the exception has existed since 8.0.0 (2019).
• Workaround. Reclaim the deletes before classifying (a merge that drops them removes the trigger), or catch IllegalArgumentException and treat the input as unclassified. Under soft-delete retention a merge keeps the retained versions (even forceMerge(1)), so there the retention query or the retained data has to change first.

The test catches only the IllegalArgumentException and turns it into a failure that states expected vs actual, including numDocs and docFreq. NoMergePolicy is just the simplest way to keep the deletes: a stock LogDocMergePolicy before a merge or one document soft-updated 4 times under SoftDeletesRetentionMergePolicy (even after forceMerge(1)) reach the same counts and fail the same way, and so does KNearestFuzzyClassifier.assignClass("apple") on the same documents with a class field added. Formatting: google-java-format reflows the fail(...) message. The test could equally be a method of the existing TestKNearestFuzzyClassifier.

TestNearestFuzzyQueryDeletedDocs > testDeletedDocsNotMergedAway FAILED
    java.lang.AssertionError: expected: 1 hit (numDocs=1, docFreq(apple)=5); actual: boost must be a positive float, got -0.098612286
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:643F8D37CED96DDE]:0)
        at junit@4.13.2/org.junit.Assert.fail(Assert.java:89)
        at org.apache.lucene.classification.utils.TestNearestFuzzyQueryDeletedDocs.testDeletedDocsNotMergedAway(TestNearestFuzzyQueryDeletedDocs.java:53)

1 test completed, 1 failed

# the same with -Ptests.asserts=false and on the 10.6.0 release jars; the exception is thrown at
#   BoostQuery.<init>(BoostQuery.java:41) <- NearestFuzzyQuery.rewrite(NearestFuzzyQuery.java:248) (:245 on 10.6.0)
# with the recommended fix the test passes

• The two counts come from different populations, and Lucene documents it. IndexReader.docFreq: "This method does not take into account deleted documents that have not yet been merged away" (IndexReader.java:402-404). TFIDFSimilarity.idfExplain: FieldStats#docCount() "is used instead of IndexReader#numDocs() because also TermStats#docFreq() is used, and when the latter is inaccurate, so is FieldStats#docCount(), and in the same direction" (TFIDFSimilarity.java:360-363). MoreLikeThis takes its idf corpus size from ir.getDocCount(fieldName) (MoreLikeThis.java:634).
• Queries must not produce negative scores. MIGRATE.md:1368-1375, "Scorer must produce positive scores (LUCENE-7996)": "If you have custom query implementations, you should make sure their score formula may never produce negative scores. As a side-effect of this change, negative boosts are now rejected".
• Fixed as a bug before, in the highlighter. LUCENE-1355 (#2432, 2008): "An index with a few documents and many deletes can report a lower total docs than docFreq for a term - total docs will account for deletes while docFreq will not - this causes the idf to be negative" (fixed by clamping docFreq). LUCENE-4289 (#5357, 2012): "highlighter uses numDocs instead of maxDoc" (fixed by reader.numDocs() to reader.maxDoc(), which also removed the clamp). QueryTermExtractor.java:74 and WeightedSpanTermExtractor.java:549 use reader.maxDoc() today. NearestFuzzyQuery (2017) copied the older pairing from the sandbox FuzzyLikeThisQuery.
• No precondition anywhere. No javadoc in the classification module restricts the classifier or the query to merged or deletion-free indexes, and Classifier.assignClass promises to "Assign a class (with score) to the given text String" (Classifier.java:31). The shared test base starts from deleteAll() (ClassificationTestBase.java:243, which keeps no deleted documents), then only adds documents, commits and calls forceMerge(1) (:231-232, :259-260), so the case was never exercised.
• History. The LUCENE-7838 discussion (#8889) is about removing a sandbox dependency and never mentions idf or deletes; the LUCENE-7996 commit did not touch the classification module, so nothing shows its new rule was checked against this formula. Before NearestFuzzyQuery, the first form of KNearestFuzzyClassifier (bd9e32d3583, 2017-05-18) used the sandbox FuzzyLikeThisQuery with the same pairing; no release shipped it (7.0.0 already has NearestFuzzyQuery).

The strongest case for "works as designed": the classifier is @lucene.experimental; the docFreq skew is documented behaviour; the formula was carried over from FuzzyLikeThisQuery; the trigger needs more than 63% of the index deleted, which the default merge policy avoided in every test; and BoostQuery rejects negative boosts on purpose. A maintainer could also prefer a clamp, or reject a change of ranking on indexes with deletes.

Why it fails: the tag's text, "This API is experimental and might change in incompatible ways in the next release", warns about API changes, not about exceptions on valid indexes, and NearestFuzzyQuery is not tagged; the documented docFreq skew is exactly why numDocs is the wrong partner (core similarities and MoreLikeThis avoid it, and the highlighter was fixed for it twice); the commit made a trimmed copy, and neither it nor its discussion mentions idf or deletes, so nothing shows the mismatch was chosen; rarity lowers the impact but does not make supported merge policies illegal; and the BoostQuery check is correct: the defect is the formula that feeds it a negative number, which MIGRATE.md forbids. The earlier cross-model reviews rated the intended-behaviour case weak, the fact check's cross-model reviewer moderate, without doubting the bug. Verdict: not intended; a small bug.

Options A and B were applied in a scratch worktree of 645d4fe14a20 and run with gradle (the worktree was removed after the runs); option C was not built. Any fix changes the idf, and so the order of neighbours, on indexes with deletes. The sandbox FuzzyLikeThisQuery has the same numDocs() line (FuzzyLikeThisQuery.java:204, read, not executed), and open draft PR #16847 edits the same lines (int to long) without touching this bug, so a fix PR may need a rebase.

One line at NearestFuzzyQuery.java:140; the comment is optional. The same change LUCENE-4289 made in the highlighter.

// docFreq also counts deleted documents that are not merged away yet, so take the corpus
// size from the same population: then docFreq <= corpusNumDocs and idf stays >= 1
int corpusNumDocs = reader.maxDoc(); // was: reader.numDocs()

Trade-offs

• docFreq ≤ maxDoc, so idf = ln((maxDoc + 1) / (docFreq + 1)) + 1 ≥ 1, for the variant-average fallback too. On an index without deletions numDocs() == maxDoc(), so nothing changes there.
• On an index with deletions the idf values, and so the neighbours' order, change, in line with core scoring statistics, which also count deleted documents.

What the checks validated

• Gradle, stock 645d4fe14a20: the repro fails with assertions on and off.
• Gradle with the fix (seed DEADBEEF): the repro and every test of the classification module, 44 tests (the repro plus 43 upstream, including TestKNearestFuzzyClassifier 3), 1 skipped (Test20NewsgroupsClassification, @Monster), 0 failures, with assertions on and off. checkGoogleJavaFormat accepts the change.
• Earlier hunt checks: the finder's, a verifier's, the adjudication's and the minimization's shadowed fix checks pass, and all 7 checks of the tiebreak program pass with it on main.

int corpusNumDocs = terms.getDocCount(); (terms is the field's Terms from MultiTerms.getTerms(reader, field) at NearestFuzzyQuery.java:133: the leaf's own Terms for one segment, a MultiTerms for several), as TFIDFSimilarity and MoreLikeThis do.

int corpusNumDocs = terms.getDocCount(); // was: reader.numDocs()

Trade-offs

• A term's docFreq in the field is at most the number of documents with the field (both count deleted documents), so the idf stays ≥ 1.
• Also changes the idf on deletion-free indexes whenever the field is sparse (docCount < maxDoc), so it moves more rankings than A.

What the checks validated

Same scratch worktree, gradle, seed DEADBEEF, assertions on: the repro and the classification module, 44 tests, 1 skipped, 0 failures, with the codecs those runs drew. MoreLikeThis keeps a fallback to numDocs() for getDocCount() == -1 (MoreLikeThis.java:635-637) that B lacks; at 645d4fe14a20 the Terms.getDocCount() javadoc documents no -1 case (Terms.java:117-121).

The LUCENE-1355 style: df = Math.min(df, corpusNumDocs) before computing the idf.

Trade-offs

Never negative, but it keeps mixing the two populations for every term below the clamp; LUCENE-4289 later replaced exactly this clamp with maxDoc() in the highlighter.

Method: non-formal baseline (coverage-guided fuzzing).

• The hunt unit for the classification module ran Jazzer 0.30.0 on a property harness that builds random small training sets (with deletions under NoMergePolicy) and checks invariants for every classifier: 2 runs of 20 minutes, 427,743 executions. Its one finding, after 184,134 runs, was IllegalArgumentException: boost must be a positive float, got -0.25276297 from NearestFuzzyQuery.rewrite via KNearestFuzzyClassifier.getClasses. The seeded random driver (10,000 cases) had not hit it.
• The value ln(2/7) + 1 gave away docFreq 6 against numDocs 1, and the finder wrote a public-API JUnit on the stock jar. Independent verifiers then reproduced it with their own tests and other triggers, found the exact threshold (docFreq 5 for 1 live document) and that the exception dates from 8.0.0; the case was later reduced to the one-method test.