Sequential DrillSideways skips the exact check of the second-cheapest drill-down dimension on segments without deletions

Sequential DrillSideways (no executor, the default) picks one of three scoring strategies by cost. In the "drill-down advance" strategy, DrillSidewaysScorer tests the second-cheapest drill-down dimension with acceptDocs == null || acceptDocs.get(docID) && (dc.twoPhase == null || dc.twoPhase.matches()) (DrillSidewaysScorer.java:396-397). Java's && binds tighter than ||, so on a segment without deletions TwoPhaseIterator#matches() is never called and every document in that dimension's approximation counts as matching it. Such a document becomes a drill-down hit if it matches the base query and every other dimension, or a near-miss counted in one other dimension's sideways facets if it misses exactly that dimension: in the repro, brand = acme and price in [0, 10] (a LongRange#getQuery with a fast-match point range [0, 100]) over 200 documents gives hits [0, 1] instead of [0] and brand sideways counts {acme=2, other=2} instead of {acme=1, other=1}, while searcher.count(ddq) and concurrent DrillSideways are correct. It needs a two-phase drill-down (range or distance with a fast-match query, phrase, slow doc-values query) that sorts second by cost and costs under a tenth of the base query, once the query-first rule is rejected, for example with browse-all. Introduced in 5.3.0 (LUCENE-6648, 2015); every release since and main.

• DrillDownQuery: a base query plus drill-down dimensions; "all dimensions are AND'd against each other and the base query" (DrillDownQuery.java:112-114).
• DrillSideways: runs a DrillDownQuery and also computes, per dimension, the "sideways" counts: documents that match everything except that dimension, "so that a dimension does not disappear after the user drills down into it" (DrillSideways.java:50-52). Without an ExecutorService it uses the sequential ("standard") implementation (DrillSideways.java:352-356).
• Two-phase query: a query whose iterator first proposes cheap candidates (the approximation) and then confirms each with TwoPhaseIterator#matches(). LongRange#getQuery(fastMatchQuery, valueSource) proposes the fast-match query's documents and confirms each by reading its value (range/LongRange.java:155-192, :322-324).

1. DrillSidewaysQuery sorts the dimensions by approximation cost, cheapest first; a dimension with no scorer in the segment becomes an empty iterator of cost 0 (DrillSidewaysQuery.java:157-161, :181-182).
2. DrillSidewaysScorer#score picks the strategy (DrillSidewaysScorer.java:100-147): query-first if scoreSubDocsAtOnce or base cost < sum of dimension costs / 10; else drill-down advance if there are at least 2 dimensions and dims[1] cost < base cost / 10; else union. Repro: 200 < 6 / 10 is false, 4 < 200 / 10 is true.
3. IndexSearcher passes the segment's live docs as acceptDocs; they are null when the segment has no deletions (IndexSearcher.java:851, ScorerUtil.java:91-93).
4. In doDrillDownAdvanceScoring the dims[0] loop checks deletions and matches() separately (:370, :373). The dims[1] loop writes acceptDocs == null || acceptDocs.get(docID) && (dc.twoPhase == null || dc.twoPhase.matches()) (:396-397), which Java reads as acceptDocs == null || (get && matches). With acceptDocs == null every candidate passes unchecked and is recorded as matching dims[1] (:399-428).
5. The base query is then folded in with real matches (:441-473) and dims[2..] call matches() (:495). At collection a document matching the base and every dimension is a hit, one missing exactly one dimension is a near-miss of it (:517-531). So, with two dimensions and a browse-all base as in the repro, a document in dims[0] and only in dims[1]'s approximation becomes a hit, and one only in dims[1]'s approximation becomes a near-miss of dims[0], counted in dims[0]'s sideways facets.
6. Every other check of a dimension calls matches(): query-first (:181, :287, :304), union (:620, :645), dims[0] and dims[2..]. With deletions the expression reduces to get && matches, which is correct.

A trace of the sequential search (a print-only copy of DrillSidewaysScorer) shows acceptDocsNull=true, costs 200 / 2 / 4, strategy=drillDownAdvance, matchesCalled=false for docs 0-3, docs 0 and 1 collected as hits and docs 2 and 3 as near-misses of brand.

• Who is exposed. Applications that run DrillSideways without an ExecutorService on a DrillDownQuery with at least two dimensions, one added through DrillDownQuery#add(String, Query) with a two-phase query: LongRange/DoubleRange#getQuery(fastMatchQuery, valueSource) (the facet module's range and distance drill-down; DistanceFacetsExample builds one with a bounding-box fast-match query, though with a single dimension, so the demo itself is not affected), a PhraseQuery, SortedSetDocValuesField.newSlowExactQuery, NumericDocValuesField.newSlowRangeQuery, or any other query with a two-phase iterator.
• Exact trigger. The two-phase dimension is dims[1]: second-cheapest by approximation cost. A dimension whose value is absent from the segment gets an empty cost-0 iterator and sorts first, so the two-phase dimension is then dims[1] if it is the cheapest of the rest (automatic with two dimensions; executed by an independent reproducer); with more than one such dimension the segment gets no scorer (DrillSidewaysQuery.java:171-179). Its cost is below a tenth of the base query's, query-first is not chosen, scoreSubDocsAtOnce() is false (the default), and the segment has no deletions (append-only, or freshly merged).
• Illustration of the cost rule (arithmetic, not run): a 1,000,000-document segment, browse all, a "within 10 km" drill-down whose bounding box holds 50,000 documents and a brand drill-down with 10,000. 1,000,000 < 6,000 is false and 50,000 < 100,000 is true, so drill-down advance runs with the distance dimension as dims[1].
• Default or opt-in. Sequential mode is the default of the 3- and 4-argument constructors; default codec, IndexWriterConfig and searcher. The two-phase drill-down comes through the method marked "Expert: add a custom drill-down subQuery", and DrillSideways is @lucene.experimental. Ordinary DrillDownQuery#add(dim, path...) drill-downs are TermQuery and are not affected.
• Masking on 9.x and 10.x. The default LRUQueryCache there can cache a frequently used drill-down on segments of 10,000 or more documents; a cached drill-down has no two-phase iterator, so the effect depends on the cache state. The segment size is necessary, not sufficient: the default caching policy also wants to have seen the query 2 to 5 times, depending on its type, and the cache's size and cost limits apply. Main disables the query cache by default (GITHUB#14187).
• Downstream. Elasticsearch, OpenSearch and Solr do not use DrillSideways (read-only code search), so this reaches applications that embed the facet module directly.
• What you see. No error. Extra hits in TopDocs and the hit count, inflated drill-down facet counts and inflated sideways counts of the other dimensions. The extra documents sit just outside the requested range, radius or phrase, so they are easy to miss. The hit count and facet counts never go down, but a false hit competes for the same bounded top-N (DrillSideways.java:227-234), so it can push a true hit out of the returned page, and an inflated label can push another out of a top-K facet list. No hang, crash or index change.
• Frequency. Deterministic once the shape is present: the repro failed 10 of 10 iterations. No prevalence was measured.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<[0]> but was:<[0, 1]>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:120)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.facet.TestDrillSidewaysTwoPhaseSecondDim.testPhraseDrillDownSequential(TestDrillSidewaysTwoPhaseSecondDim.java:73)

• Contract. DrillDownQuery#add: "all dimensions are AND'd against each other and the base query" (DrillDownQuery.java:112-114). DrillSidewaysResult#drillDownFacetsCollector: "populated based on hits that match the full DrillDownQuery, treating all drill down dimensions as required clauses"; #drillSidewaysFacetsCollector: "Each collector exposes the hits that match on all DrillDownQuery dimensions, but treating their corresponding sideways dimension as optional" (DrillSideways.java:265-266, :272-274). The class javadoc presents the standard and concurrent implementations as alternatives that differ in speed and duplicate work, not results (DrillSideways.java:57-63). Nothing restricts the "Expert" subquery (DrillDownQuery.java:121-125).
• The code's own intent. The same method checks matches() for dims[0] (DrillSidewaysScorer.java:373) and dims[2..] (:495); query-first and union always do (:181, :287, :304, :620, :645). Only the missing parentheses make the dims[1] check depend on deletions.
• Upstream tests. TestDrillSideways#testRandom adds newSlowExactQuery drill-downs: "sometimes use a two-phase approach to ensure code coverage" (TestDrillSideways.java:1283-1291). Each document has no value for a dimension only with probability 4/25 (the first value is dropped with probability 1/5 and a second one is still added with probability 1/5; :1086, :1103), so a two-phase dims[1] practically never costs under a tenth of the base (read from the cost distribution, not an instrumented run). With the fix all 307 facet tests pass, so no test pins the current behaviour.
• History. LUCENE-6648 (#7706, @jpountz, +1 by @mikemccand, 2015-07-01) replaced Filters with two-phase iteration and appended && (dc.twoPhase == null || dc.twoPhase.matches()) to the existing acceptDocs == null || acceptDocs.get(docID) without parentheses; its issue comment calls it "a simple 1:1 migration", and the same comment names Range.getQuery(fastMatchQuery, ValueSource) as code "explicitely split into a fast approximation and a slow confirmation", the shape of this doc's range test. Before it, custom drill-downs were consumed through Scorer.nextDoc(), which returns only verified matches. The finder's PhraseQuery test (one 5.x-API source) passes on 5.2.1 and fails on 5.3.0, where the parentheses alone make it pass.
• Precedent. #12211, fixed by #12212 (2023), treated a two-phase mistake in this scorer as a bug: a two-phase base query in the query-first strategy missed hits.

The strongest case for "intended or misuse" (built by the devil's advocate and by a reviewer from another model family, OpenAI GPT-5.6, reading only): DrillSideways is @lucene.experimental, add(String, Query) is "Expert", the drill-down-advance scorer is a package-private performance specialisation, the LongRange#getQuery javadoc itself suggests a PointRangeQuery for a static field, and nobody has reported this since 5.3.0 (2015).

Why it fails: experimental allows API changes, not wrong results, and nothing restricts the subquery. The module's own range and distance drill-downs are two-phase, and the javadoc's PointRangeQuery advice is about speed for static fields; computed values such as distances need the fast-match form, and phrases and slow doc-values queries fail the same way. Results that depend on an internal cost estimate and on whether a segment has deletions cannot be a contract, and concurrent mode and plain searches agree with the javadoc. The reviewer from another model family rated the intended case weak; the devil's advocate found no argument that survives. The experimental and Expert markers remain a weak argument for limited compatibility promises (and so for how far a fix is backported), but none of them documents approximation-only matching.

Option A was validated in a scratch worktree at ea57360912ed (removed afterwards); B and C were not run. Until a fix ships, users can avoid the bug with an executor or scoreSubDocsAtOnce() (option D, executed).

Make the deletions check and the two-phase check two conjuncts, as the dims[0] loop effectively does.

-        if (acceptDocs == null
-            || acceptDocs.get(docID) && (dc.twoPhase == null || dc.twoPhase.matches())) {
+        if ((acceptDocs == null || acceptDocs.get(docID))
+            && (dc.twoPhase == null || dc.twoPhase.matches())) {

Trade-offs

On deletion-free segments the loop now calls matches() once per dims[1] candidate: the same check it already applies to every live candidate on segments with deletions, and what every other loop does. Queries that hit the bug get slower by that cost and correct. No benchmark was run.

What the checks validated

• Through gradle at ea57360912ed: the repro fails on stock (assertions on and off) and passes with the fix.
• With the fix: TestDrillSideways* 15 tests (the 14 upstream plus the new one), 0 failures; :lucene:facet:test :lucene:demo:test 307 facet tests including the new one (7 skipped) and 23 demo tests, 0 failures, assertions on; :lucene:facet:test with assertions off 307 tests, 0 failures (8 skipped); :lucene:facet:check -x test (checkGoogleJavaFormat, ecjLint, forbiddenApis) successful.
• Earlier steps: the finder's, the independent reproducer's (7 of 7) and the devil's advocate's tests pass with the same patch; on 5.3.0 the parentheses alone make the repro pass; the bounded-exhaustive harness finds 0 mismatches in 84,278,820 checks with it.

if (acceptDocs == null || acceptDocs.get(docID)) { if (dc.twoPhase == null || dc.twoPhase.matches()) { ... } }, the shape of DrillSidewaysScorer.java:370-373.

Trade-offs

Same semantics as option A with a larger diff (re-indented body). Not run.

#11076 (LUCENE-10037, open) explores a single scoring implementation; a 2023 comment there commented out both doUnionScoring and doDrillDownAdvanceScoring and reported a drill-sideways benchmark run with no statistically significant difference (most point estimates slightly negative, within the noise); the maintainer replied that they were not sure that benchmark could be trusted to detect such a change.

Trade-offs

Removes the buggy loop, but it is a design change that needs its own benchmarks and discussion; not suitable as a bug fix or backport. Not run.

Pass an ExecutorService (concurrent DrillSideways), or subclass DrillSideways and return true from scoreSubDocsAtOnce(), which forces query-first scoring.

Trade-offs

Concurrent mode duplicates work per dimension (class javadoc); scoreSubDocsAtOnce() warns of "some performance loss".

What the checks validated

Both give the correct result on stock main: the repro's concurrent branch passes, and the repro with scoreSubDocsAtOnce() returning true passes.

Method: bounded-exhaustive translation validation, a hunt unit (facets, taxonomy, SSDV, DrillSideways), 2026-10-26.

• The finder read DrillSidewaysScorer while designing a harness that drives the real DrillSidewaysQuery and DrillSidewaysScorer with set-backed base and drill-down queries (optional two-phase iterators), a live-docs overlay, each of the three strategies forced, documents at the 2048-document chunk boundaries (positions 0, 2047, 2048, 4095), and compares every hit and sideways set with an exact set-semantics reference.
• On the stock main jars: 310,464 of 43,808,304 checks (4 documents, 2 dimensions) and 248,950 of 36,015,000 (3 documents, 3 dimensions) mismatched, every one in the drill-down-advance strategy; with no two-phase dimensions, 0 of 3,145,728 (5 documents, 2 dimensions); with the fix, 0 mismatches in 84,278,820 checks (4,455,516 + 36,015,000 + 43,808,304; also 2 documents, 4 dimensions).
• The finder then wrote a public-API JUnit (PhraseQuery and slow doc-values second dimension). Then a refute review by an independent reviewer from another model family (OpenAI GPT-5.6, reading only; its "not real" was procedural, since it could not run code, and it raised no substantive refutation: apart from the failed execution, its only not-real item was the experimental/Expert scope argument, which it said documents no relaxed matching), an independent reproducer, a GPT-5.6 "works as designed" review with executed re-runs, a devil's advocate with its own range variant, an adjudication, a history sweep from 5.2.1 to main, a prior-report search, a security assessment, a registry re-run and a credibility audit. No cross-model review that executes code has run.