k-NN classifiers score their training filter query, so a filter that removes nothing can change the class

KNearestNeighborClassifier, KNearestNeighborDocumentClassifier, KNearestFuzzyClassifier and BooleanPerceptronClassifier take a training query, documented as "a Query to eventually filter the docs used for training the classifier, or null if all the indexed docs should be used". The k-NN classifiers add it, and the class-field restriction WildcardQuery(class:*), to their similarity search as scoring Occur.MUST clauses, so the filter's score is added to every neighbour's score; each vote is then weighted by score / top score. A filter that keeps every document can therefore still change the class scores (from 5.3.0 for constant-score filters), and the class whenever the shift outweighs the similarity gap: on three documents, MatchAllDocsQuery or a year range turns Y (Y 0.333, X 0.309) into X (X 0.417, Y 0.333). A filter whose score varies between documents can also change which documents are the nearest neighbours, and the perceptron, which trains in the score order of the same kind of query, can learn a different model. The filter still restricts which documents are used. Every release from 4.6.0 to 10.5.2 and main is affected; Occur.FILTER for the training query fixes it, and new BoostQuery(q, 0f) (same matches, no score) is a workaround.

• Training query: a constructor argument of KNearestNeighborClassifier, KNearestNeighborDocumentClassifier, KNearestFuzzyClassifier and BooleanPerceptronClassifier: "a Query to eventually filter the docs used for training the classifier, or null if all the indexed docs should be used" (KNearestNeighborClassifier.java:82-83; the same in the other three). For example, train only on the training split, one tenant or one time window.
• k-NN vote: the classifier searches for the k labelled documents most similar to the input (MoreLikeThis, or NearestFuzzyQuery for the fuzzy variant). Each neighbour votes for its class with weight score / maxScore, its score divided by the top hit's score (KNearestNeighborClassifier.java:195, :207); a class's score is the sum of its neighbours' weights divided by k (:219-225).
• MUST and FILTER: in a BooleanQuery, a MUST clause must match and adds its score; FILTER is "Like MUST except that these clauses do not participate in scoring" (BooleanClause.java:35).

1. knnSearch adds the similarity query as SHOULD (MUST in the fuzzy classifier, so there a neighbour must match at least one fuzzy term), then WildcardQuery(class:*) as MUST and the training query as MUST (KNearestNeighborClassifier.java:171-178; KNearestFuzzyClassifier.java:141-145; KNearestNeighborDocumentClassifier.java:144-151).
2. So every hit's score is its similarity plus 1.0 for the class-field clause plus the training query's score. MatchAllDocsQuery and point ranges add a constant 1.0; a term on a text field adds a BM25 score that varies with field length.
3. The vote divides each score by the top score. Adding the same number to every score pulls every weight towards 1, which favours the class with more, weaker neighbours; a varying addition can also reorder the top k.
4. BooleanPerceptronClassifier builds the same two MUST clauses and trains over the hits in score order (BooleanPerceptronClassifier.java:130-140). Without a training query every hit scores 1.0 (doc-id order), and a constant-score filter keeps that order; a text-scored filter can reorder the training. The weights change only for a document the current model misclassifies (:155), and with batch size 1 each change takes effect at once (:162), so the model depends on the order.

• k = 3, documents "aa bb" (Y), "aa" (X), "aa" (X), input "aa bb bb". Similarity: "aa bb" 1.3037, each "aa" 0.0676. With the class-field +1.0: 2.3037 and 1.0676, weights 1.0 and 0.4634, so X = 2 x 0.4634 / 3 = 0.3090 < Y = 0.3333. With MatchAllDocsQuery (another +1.0): 3.3037 and 2.0676, weights 1.0 and 0.6258, so X = 0.4172 > Y = 0.3333. An IntPoint year range over all three documents gives the same numbers; KNearestNeighborDocumentClassifier flips the same way; KNearestFuzzyClassifier keeps X, 0.4997 becomes 0.5554.
• k = 1, documents "aa" (X, tags "train news sports 2024") and "aa bb" (Y, tags "train"), input "aa", training query TermQuery(tags:train). Similarity: "aa" 0.0960, "aa bb" 0.0729. The tag term scores 0.1098 on the one-word tags field and 0.0665 on the four-word one, so "aa bb" ranks first (1.1828 against 1.1625) and the answer changes from X to Y.
• BooleanPerceptronClassifier (batch size 1, bias 1.5) on two "aa" documents labelled true (tags "train pad") and false (tags "train"): "aa" is false without and true with TermQuery(tags:train).
• Control everywhere: new BoostQuery(q, 0f) matches the same documents with score 0, and for these filters, which keep every document, it gives exactly the no-query answer, so the cause is the filter's score, not its matches.
• Secondary: with no training query at all, the class-field clause's 1.0 still gives neighbours that share no term with the input a non-zero vote (in the text and document classifiers; the fuzzy classifier requires a fuzzy term match). The vote formula is not documented, so this is a side effect of the same MUST clauses, not a contract break.

• Who is exposed. Users of the opt-in lucene/classification module (the four classes are @lucene.experimental) who pass a training query, for example to train on one split, tenant or time window. Callers that pass null get only the secondary class-field effect.
• How often. On a 300-document, 3-segment index (283 live) with four filters that select the same 168 documents, the class list differed from the zero-score control in 590 to 705 of 800 cases per classifier and filter (a case is one of 5 values of k and one of 160 input texts; each calls getClasses and assignClass). The assigned class changed in 0 of 800 cases for the two constant-score filters (a StringField term, an IntPoint range) and in 40 to 320 of 800 for the two whose score varies (a text-field term, a text-scored BooleanQuery). That probe ran the three k-NN classifiers on main only. In the finder's bounded-exhaustive run over 3,717,152 tiny cases, MatchAllDocsQuery changed the KNearestNeighborClassifier scores in 2,185,012 cases and the class in 15,617; an all-matching text-field term changed the class in 31,412.
• Scale. None needed: three documents flip the class. The shift depends on the filter's score next to the similarity scores: MatchAllDocsQuery and point ranges add 1.0, while a StringField term every document has adds very little (a verifier saw X 0.31893 become 0.32096, no flip).
• Silent. Nothing fails, and there is no unfiltered baseline to compare with; the scores look plausible.
• Not affected. Which documents are used: in 9,600 filtered cases (19,200 filtered calls) there were 0 neighbours from outside the filter and 0 deleted documents. The naive-Bayes classifiers' MUST sites only count hits. BM25NBClassifier adds the training query as MUST too, and a verifier's probe shows MatchAllDocsQuery moving its class scores (Y 0.680 / X 0.320 to Y 0.561 / X 0.439); it is not part of the confirmed claim.
• Versions. Every release from 4.6.0 (2013) to 10.5.2 and main; constant-score filters matter from 5.3.0.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.junit.ComparisonFailure: expected:<[Y]> but was:<[X]>
       org.junit.Assert.assertEquals(Assert.java:117)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.classification.TestKnnTrainingQueryIsScored.testMatchAllTrainingQueryChangesAssignedClass(TestKnnTrainingQueryIsScored.java:49)

• Contract. KNearestNeighborClassifier.java:82-83: "@param query a Query to eventually filter the docs used for training the classifier, or null if all the indexed docs should be used"; the field javadoc (:69-72): "a Query used to filter the documents that should be used from this classifier's underlying LeafReader". The same wording in KNearestFuzzyClassifier.java:65, :79, KNearestNeighborDocumentClassifier.java:58 and BooleanPerceptronClassifier.java:72. Read as a filter, a query that keeps every document should give the null result. The javadoc does not say in so many words that the score is ignored, so this is the natural reading of the contract rather than its literal text; Lucene's CHANGES entries say the same about filtering in general: LUCENE-6227 added FILTER "to filter documents without participating in scoring (on the contrary to MUST)" (lucene/CHANGES.txt:10935-10936, 5.1.0), and LUCENE-6301 asks for "the BooleanClause.Occur.FILTER clause in order to let Lucene know that a Query should be used for filtering but not scoring" (:9931-9934, 5.4.0).
• Tests. TestKNearestNeighborClassifier#testBasicUsageWithQuery (:145-159) and the matching fuzzy, perceptron, naive-Bayes and document tests check only the assigned class on one sample index. No test compares a filtered and an unfiltered run, and both fixes keep the whole module suite green, so no test pins the scored behaviour.
• History. LUCENE-5284 (#6348, commit e60f7af1d16, 2013-10-17, "added method for training with a Query filter") added the query as a MUST clause; Occur.FILTER arrived only with LUCENE-6227 (a70006f2f5f, 2015-02-10, 5.1.0). LUCENE-5338 (#6402, debb363f6e3, "avoid considering unlabeled documents for training") added the class-field clause, also as a filter. LUCENE-6654 (#7712, 35810f4c77d, 5.3.0) added the score / maxScore vote because "the MLT score the documents accordingly" and tested only with a null query. The document (2015) and fuzzy (2017) classifiers copied the clauses; no commit ever used FILTER in the module.

The strongest "intended" case: a Query normally scores and MUST keeps the score on purpose; FILTER already existed when LUCENE-6654 added the score-weighted vote and kept MUST; later classes copied the pattern; BM25NBClassifier uses a scored MUST clause (the class term) as its signal; the module is @lucene.experimental; and a zero-boost workaround exists. The cross-model reviewer rated this argument moderate in both of its lenses. The later fact check by a reviewer from the same family adds: callers may have used the query's score as a secondary ranking signal, so a fix changes results that have shipped since 4.6.0; the class-field constant gives every admitted neighbour a vote, much like the count vote before 5.3.0; and an online perceptron depends on example order anyway.

Why the bug reading is still the stronger one for the k-NN classifiers: every javadoc and the introducing issue call the parameter a filter, with null meaning "all the indexed docs", Lucene's CHANGES entries tell users to use FILTER when a query should filter but not score, and nothing anywhere says its score should shift the similarity or the votes (an absence, not proof). The MUST clauses predate FILTER and were carried over unchanged; no commit, issue or comment discusses the choice, and LUCENE-6654 attributes the ranking to the MoreLikeThis score. An ordinary year-range filter that keeps every document flips the class. The BoostQuery(q, 0f) workaround is undocumented; for such a filter it gives exactly the unfiltered result. @lucene.experimental allows API changes, not results that contradict the javadoc. Narrower points stand: the javadoc never says in so many words that the score is ignored; the class-field constant is a weaker finding because the vote formula is undocumented; and the perceptron's training order is undocumented. Every reviewer rated the bug real; the adjudication confirmed it with high confidence. The fact-check reviewer rated the intended case moderate for the training query and the class field and strong for the perceptron order.

The training query still decides which documents can be neighbours or training examples; it just adds no score. Fix B leaves the class-field clause as a scoring MUST, so maxScore stays at least 1 and no division by zero can appear; fix A, the hunt's original proposal, also turns that clause into FILTER, which changes results for callers without a training query too. Until a fix ships, pass new BoostQuery(query, 0f) as the training query: it stays a required clause that matches the same documents, with score 0, so it acts as a non-scoring filter (for a filter that keeps every document, exactly the unfiltered result). Nothing is stored wrong, so no reindex is needed. Fix B changes results for callers that pass a training query, so it needs a CHANGES entry. Either way, add a test that a filter keeping every document gives the null result for each classifier. BM25NBClassifier has the same training-query MUST (BM25NBClassifier.java:216, :227); if it is included, change only those clauses, because its class TermQuery score is the intended signal there.

KNearestNeighborClassifier.java:177, KNearestNeighborDocumentClassifier.java:151, KNearestFuzzyClassifier.java:145, BooleanPerceptronClassifier.java:135: MUST becomes FILTER.

// KNearestNeighborClassifier:177 and KNearestNeighborDocumentClassifier:151
-      mltQuery.add(query, BooleanClause.Occur.MUST);
+      mltQuery.add(query, BooleanClause.Occur.FILTER);
// KNearestFuzzyClassifier:145
-      bq.add(query, BooleanClause.Occur.MUST);
+      bq.add(query, BooleanClause.Occur.FILTER);
// BooleanPerceptronClassifier:135
-      q.add(new BooleanClause(query, BooleanClause.Occur.MUST));
+      q.add(new BooleanClause(query, BooleanClause.Occur.FILTER));

Trade-offs

Implements the natural, non-scoring reading of the documented filter with the smallest change, and changes nothing without a training query. Callers that pass one get different scores, neighbours and sometimes classes than current releases give (the scored behaviour has shipped since 4.6.0), so it needs a CHANGES entry. It leaves the secondary class-field constant as it is (the independent reproducer's class-field test still fails with it), and the perceptron's initial weights and default bias still come from the whole index (BooleanPerceptronClassifier.java:99-124).

What the checks validated

• Scratch worktree of a9779ef3c193, gradle, JDK 25: the new test fails 4 of 5 on stock with assertions on and off; with the fix, the new test plus TestKNearestNeighborClassifier, TestKNearestFuzzyClassifier, TestKNearestNeighborDocumentClassifier and TestBooleanPerceptronClassifier pass (5 suites, 21 tests); the whole :lucene:classification:test passes with assertions on and off (13 suites, 48 tests, 1 skipped, 0 failures each); :lucene:classification:check -x test (google-java-format, ecj lint, forbidden APIs) passes.
• Standalone runner with the fix shadowed: the new test 5 of 5 with -ea and -da; the finder's test 4 of 4; the independent reproducer's test 7 of 8 (only its class-field case fails, as expected).

Occur.FILTER for WildcardQuery(class:*) and the training query in the four classes, plus double singleBoost = maxScore > 0 ? scoreDoc.score / maxScore : 1d; in KNearestNeighborClassifier.java:207 and KNearestFuzzyClassifier.java:172: with FILTER, when no neighbour shares a selected term (text and document classifiers), maxScore is 0 and the weights would be 0/0 = NaN. The class-field change matters differently per class: in the text and document classifiers it stops neighbours with no shared term from voting, in the fuzzy classifier (whose NearestFuzzyQuery stays MUST) it only removes the constant 1.0, and in the perceptron it changes nothing, because a constant score does not change the training order.

-    mltQuery.add(new BooleanClause(classFieldQuery, BooleanClause.Occur.MUST));
+    mltQuery.add(new BooleanClause(classFieldQuery, BooleanClause.Occur.FILTER));
     if (query != null) {
-      mltQuery.add(query, BooleanClause.Occur.MUST);
+      mltQuery.add(query, BooleanClause.Occur.FILTER);
     }
...
-          double singleBoost = scoreDoc.score / maxScore;
+          double singleBoost = maxScore > 0 ? scoreDoc.score / maxScore : 1d;

Trade-offs

• Also removes the class-field constant, so the vote becomes pure similarity weighting: in the lead case the no-query result becomes Y 0.3333, X 0.0346 instead of X 0.3090.
• Changes class scores for every caller, including those without a training query, and sometimes the class: on the lead data the fuzzy classifier with no training query answers Y (0.3333 against X 0.3321) instead of X (0.4997).
• The maxScore > 0 fallback (an unweighted vote) is a design choice for maintainers; neighbours with score 0 still fill the top k.

What the checks validated

Scratch worktree, gradle: the whole :lucene:classification:test passes with assertions on and off (13 suites, 48 tests, 1 skipped, 0 failures each) and :lucene:classification:check -x test passes (its forbiddenApisTest task was reported up to date from the fix B run; the other check tasks ran). Standalone runner: the new test 5 of 5 with -ea and -da, the independent reproducer's test 8 of 8; earlier the finder's test 4 of 4, the 42 upstream classification tests and the security probe (0 violations, 0 non-finite scores) with the same sources.

Pass new BoostQuery(query, 0f) (or new BoostQuery(new ConstantScoreQuery(query), 0f)) as the training query.

Trade-offs

It keeps the query's matches and adds no score, the same effect as fix B; a selective query still removes the documents it does not match. Every executed control used a filter that keeps every document, and there it gave exactly the unfiltered result. Every caller has to know about it; it is undocumented and not a fix.

Method: semi-formal, bounded-exhaustive translation validation: the real classifiers against an independently written reference, over every case in a small bounded space.

• The reference is a weighted k-NN over the MoreLikeThis or NearestFuzzyQuery similarity alone. Two metamorphic relations: training query MatchAllDocsQuery against none, and a text-field TermQuery that every document matches against none.
• Scope: a 4-word vocabulary; every training set of 1 to 3 documents over 2 classes (2-document sets also with each document deleted); 3,000 seeded sets of 4 to 6 documents over 2 or 3 classes; every query text of 1 to 3 words; k = 1 to 4: 3,717,152 cases per classifier.
• KNearestNeighborClassifier: MatchAllDocsQuery changed the scores in 2,185,012 cases and the class in 15,617; the all-matching term changed the class in 31,412. The document variant equalled the text variant in every case.
• With fix A, a reduced run of 527,680 cases showed 0 differences of any kind: the independently written reference equals the fixed code.
• The finder then wrote the 4-test JUnit; the independent reproducer, a cross-model reviewer, a devil's advocate and an adjudication each re-ran or rewrote it.