A point range that covers every indexed point also matches documents without points on a MultiReader (10.3.0 and later)

Since 10.3.0 (#14609), PointRangeQuery.rewrite checks whether the query range contains every indexed point of the whole reader and, if not every document has a point, replaces the query with FieldExistsQuery(field) (PointRangeQuery.java:544-572 at e1faeba08bde). The reader-wide bounds skip leaves without points (PointValues.java:150-151), and the check that should keep the rewrite safe, canRewriteToFieldExistsQuery (:578-592), refuses a leaf only when the field has no doc values, no norms and no vectors there; it never asks whether the leaf has points. On a MultiReader over separately built indexes, where one index stores the field without points (doc values only, text, a keyword or a vector), the rewritten query therefore matches every document of that index that has norms, a vector or doc values for the field, whatever its value, and under MUST_NOT it drops them. The repro counts 2 instead of 1; 10.2.2 returns 1. A single index created by Lucene 9 or later reaches it only through another defect that lets in documents with and without the field's points, such as a deleteAll() that overlaps addIndexes.

• Points and doc values: two ways to store a numeric field. Range queries such as LongPoint.newRangeQuery read points; FieldExistsQuery reads norms, vectors or doc values. LongField writes a point and a SORTED_NUMERIC doc value.
• LUCENE-9334: "A field across all documents within an index must be indexed with the same index options and data-structures" (lucene/CHANGES.txt:4771-4775). IndexWriter enforces it per index.
• MultiReader: "A CompositeReader which reads multiple indexes, appending their content" (MultiReader.java:22-25). Each sub-index becomes one or more leaves; there is no schema check across them.

1. PointRangeQuery.rewrite takes the reader-wide bounds, PointValues.getMinPackedValue(reader, getField()) and getMaxPackedValue (search/PointRangeQuery.java:552-553). Their javadoc: "Leaves that do not have points for the given field are ignored." (index/PointValues.java:150-151, :185-186). In the repro: [100, 100].
2. relate gives CELL_INSIDE_QUERY (:560). canRewriteToMatchAllQuery compares reader.maxDoc() with the point doc count (:574-576): 2 vs 1, no.
3. canRewriteToFieldExistsQuery (:578-592) returns false for a leaf only when info.getDocValuesType() == DocValuesType.NONE && !info.hasNorms() && info.getVectorDimension() == 0 (:582-588). Index 1's leaf has NUMERIC doc values, so it passes; getPointDimensionCount() is never read. The query becomes new FieldExistsQuery(field) (:563-564).
4. FieldExistsQuery matches documents with a vector field "or a field that indexes norms or doc values" (search/FieldExistsQuery.java:38-41), per leaf through norms, else vectors, else doc values (:170-219). Index 1's document has a doc value, so it matches.
5. The premise, in FieldExistsQuery's own rewrite: "This optimization is possible due to LUCENE-9334 enforcing a field to always use the same data structures (all or nothing)." (search/FieldExistsQuery.java:143-144). True per index, not across separately built indexes.

• When every document of every leaf has the field (FieldExistsQuery.java:116-163; for example the probe's one-document index 1 with a same-named KeywordField), FieldExistsQuery rewrites further to MatchAllDocsQuery; under MUST_NOT the clause becomes MatchNoDocsQuery.
• A range that only partly overlaps the indexed points is not rewritten (CELL_CROSSES_QUERY), and one that excludes them all becomes MatchNoDocsQuery (CELL_OUTSIDE_QUERY, PointRangeQuery.java:569-570); both stay correct.
• If the point-bearing index has points only (LongPoint without doc values, norms or vectors), the check refuses that leaf and the result is correct.
• Before #14609, PointRangeQuery did not override rewrite(); every leaf was matched through its own points, so a leaf without points matched nothing.

• Who. Anyone who searches a MultiReader (or another composite reader) over separately built indexes in which one field has points plus doc values, norms or vectors in one index (for example LongField), and the same field name has doc values, norms or a vector but no points in another. Realistic shapes: a field that was doc-values-only in older indexes and became a LongField later; a name reused for text, a keyword or a vector.
• Which queries. Executed reach probe, 71 one-document cases: 54 silently wrong on main (plus 2 that throw), none silently wrong with the fix (the same 8 cases throw there as on 10.2.2). LongPoint, IntPoint, FloatPoint, DoublePoint, InetAddressPoint and BinaryPoint.newRangeQuery, LatLonPoint.newBoxQuery (a box that does not cross the dateline, which is one PointRangeQuery), and the LongField / IntField / FloatField / DoubleField.newRangeQuery factories (through IndexOrDocValuesQuery). Also inside BooleanQuery and FunctionScoreQuery.boostByValue (composite decomposition harnesses: 600 of 17,366 and 69 of 2,534 query/composite checks mismatch, 0 with the fix).
• Scope for the XxxField.newRangeQuery factories. The finding covers only point-less leaves with NUMERIC or SORTED_NUMERIC doc values outside the range, and for DoubleField and FloatField also leaves that hold only norms or vectors. IndexOrDocValuesQuery requires that "Both provided queries must match the same documents" (IndexOrDocValuesQuery.java:61), which an in-range doc-values-only leaf already breaks; and LongField and IntField.newRangeQuery wrap it in IndexSortSortedNumericDocValuesRangeQuery (LongField.java:138-146), which reads numeric doc values on every leaf (IndexSortSortedNumericDocValuesRangeQuery.java:190, :528), so over a keyword, text, string or vector leaf those two throw on 10.2.2 and with the fix (on main 6 of these 8 cases give a wrong count instead). The headline repro uses plain LongPoint.newRangeQuery.
• When. Only ranges that contain every indexed point of the reader: open-ended or wide ranges, the case #14609 targets ("particularly when using time based partitioning, range queries may overlap several indexes").
• Effect. Every document of the point-less index that has norms, a vector or doc values for the field is added, or under MUST_NOT removed, whatever its value. Silent: no exception, nothing in the index changes.
• Default vs opt-in. Default IndexWriterConfig and IndexSearcher, assertions on or off. The test picks no codec (for seed DEADBEEF the test framework chose Asserting(Lucene104); 20 of 20 random seeds fail), and the reach probe, plain Java with new IndexWriterConfig(), runs on the default codec. The rare part is the reader layout, so likelihood rare (no public report was found either); priority Medium.
• Not part of this finding. The 8 loud IllegalStateException cases ("unexpected docvalues type ... Re-index with correct docvalues type") of LongField and IntField.newRangeQuery over a point-less leaf without numeric doc values, thrown by IndexSortSortedNumericDocValuesRangeQuery (DocValues.getSortedNumeric in count, :190); 10.2.2 and the fix throw on the same 8. FieldInfos.getMergedFieldInfos throwing on such readers (a separate bug). A single segment of an index first created by Lucene 8 whose documents are not all-or-nothing for the field: another root, which this fix does not change. With those documents in separate segments of such an index it is this bug, as Who is affected says (count 2 and 2 hits on main, 1 and 1 with the fix).

TestPointRangeRewriteLeafWithoutPoints (one method, extends LuceneTestCase, package org.apache.lucene.search of lucene/core tests). Index 1 gets NumericDocValuesField("p", 5000), index 2 LongField("p", 100, Field.Store.NO); both use new IndexWriterConfig() and ByteBuffersDirectory; the searcher is a plain new IndexSearcher(new MultiReader(r1, r2)). About 1.3 s in the standalone runner (a gradle run with build startup took 12-55 s).

Index 1's value can also be written as new SortedNumericDocValuesField("p", 5000), so index 1 uses the same doc-values type as LongField and the only difference is the missing points. In that form it fails on main (-ea) and 10.6.0 (-da) and passes on 10.2.2 (-da).

As a regression test the same method fits in TestPointQueries, next to testRangeQueryOptimizesRewrites.

cp $REPRO/TestPointRangeRewriteLeafWithoutPoints.java $LUCENE/lucene/core/src/test/org/apache/lucene/search/
cd $LUCENE && ./gradlew :lucene:core:test --tests org.apache.lucene.search.TestPointRangeRewriteLeafWithoutPoints -Ptests.seed=DEADBEEF

# gradle, worktree at e1faeba08bde, -Ptests.seed=DEADBEEF (same with -Ptests.asserts=false)
TestPointRangeRewriteLeafWithoutPoints > testLeafWithDocValuesButNoPoints FAILED
    java.lang.AssertionError: expected:<1> but was:<2>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:1FD6C01022C186CF]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:633)
        at org.apache.lucene.search.TestPointRangeRewriteLeafWithoutPoints.testLeafWithDocValuesButNoPoints(TestPointRangeRewriteLeafWithoutPoints.java:51)
:lucene:core:test (FAILURE): 1 test, 1  failure

# same test, standalone runner
10.2.2 -da: PASS        10.6.0 -da: FAIL expected:<1> but was:<2>
main -ea / -da: FAIL expected:<1> but was:<2>        main + fix -ea / -da: PASS

• Contracts. LongPoint.newRangeQuery: "@return a query matching documents within this range." (LongPoint.java:212). PointRangeQuery: "Abstract class for range queries against single or multidimensional points" (PointRangeQuery.java:41). MultiReader: "A CompositeReader which reads multiple indexes, appending their content. It can be used to create a view on several sub-readers (like DirectoryReader) and execute searches on it." (MultiReader.java:22-25), no schema precondition.
• A rewrite must not change results. The un-rewritten query is correct on every leaf; 10.2.2 and 9.12.3 return the right count on the same MultiReader.
• Scoped to one index by its authors. PR #14609: "handle cases where the query either fully contains or fully excludes all documents within the shard". CHANGES 10.3.0: "rewriting to MatchAllDocsQuery/FieldExistsQuery/MatchNoDocsQuery if all docs in index are contained or excluded" (CHANGES.txt:1681-1682). The review's code comments were about reusing relate(), an early return when a dimension is neither inside nor outside, and null min/max, plus the javadoc tag order and a request for a CHANGES entry; none of them mentions composite readers.
• Tests. TestPointQueries.testRangeQueryOptimizesRewrites (TestPointQueries.java:2473) uses one consistent index; no upstream test builds a MultiReader with a point-less leaf.
• Precedent. #14399 (jpountz): "While IndexWriter enforces that the whole index has the same index sort, it is possible to create a MultiReader across several indexes which have different index sorts, so this cache is incorrect." Fixed by #16434: "This fixes incorrect results when searching a MultiReader that combines indexes with different index sorts" (CHANGES.txt:763-765). In the #15083 review jpountz wrote "MultiReader allows mixing segments that have a skip index on the field and segments that don't", and DocValuesSkipper.globalMinValue returns Long.MIN_VALUE when a leaf has the field but no skipper (DocValuesSkipper.java:145-160): the guard this check lacks.

The strongest case for "intended" (built by an intended-behaviour checker): the rewrite builds on LUCENE-9334, which Lucene relies on for several optimizations; the two indexes conflict on field p, and FieldInfos.getMergedFieldInfos throws on this exact reader ("cannot change field \"p\" from doc values type=NUMERIC to inconsistent doc values type=SORTED_NUMERIC"); PointRangeQuery is @lucene.experimental (PointRangeQuery.java:52); so a mixed-schema MultiReader could be called unsupported.

Why it fails: LUCENE-9334 is documented per index. #14609 combines a reader-wide statistic (point bounds that skip point-less leaves) with a per-leaf FieldExistsQuery that also matches point-less leaves, which stretches the per-index rule across index boundaries. MultiReader documents no schema precondition and getMergedFieldInfos documents no exception (its throw is a separate bug). @lucene.experimental marks an unstable API, not wrong results, and the factories that return the query are not experimental. The extra document's value, 5000, matches [0, 1000] under no reading; with index 1 written as SortedNumericDocValuesField the indexes differ only in the missing points and it still fails. Maintainers fixed the same kind of cross-leaf assumption as incorrect results (#14399 / #16434).

Two options; the first was built and validated, the second is simpler but gives up more of the optimization. The choice is the maintainers'.

One condition in canRewriteToFieldExistsQuery (PointRangeQuery.java:582-585). A leaf with the field but no points makes rewrite fall back to super.rewrite(searcher), the normal per-leaf point query; a leaf without the field (info == null) still allows the rewrite.

       if (info != null
-          && info.getDocValuesType() == DocValuesType.NONE
-          && !info.hasNorms()
-          && info.getVectorDimension() == 0) {
+          && (info.getPointDimensionCount() == 0
+              || (info.getDocValuesType() == DocValuesType.NONE
+                  && !info.hasNorms()
+                  && info.getVectorDimension() == 0))) {
         // Can't use a FieldExistsQuery on this segment, so return false
         return false;
       }

Trade-offs

• Keeps the optimization on every consistent layout tested: a rewrite differential over consistent layouts (points only, points plus doc values or norms, sparse, multi-valued, 2-D, absent-field segments, deletions) ran 1,250,932 checks with 0 failures and the same rewrite-kind histogram as stock (FieldExistsQuery 4,696, MatchAllDocsQuery 2,370).
• On a mixed composite the shortcut is lost for that query, which then runs as before #14609.
• It does not cover a single 8.x-created segment whose documents are not all-or-nothing; that is a different root and needs its own fix. With such documents in separate segments the fix does apply (count 2 -> 1).

What the checks validated

• Gradle in a scratch worktree at e1faeba08bde, JDK 25, -Ptests.seed=DEADBEEF (2026-11-02): the repro fails without the fix (asserts on and off) and passes with it (asserts on and off).
• With the fix: TestPointQueries (56 tests, 2 skipped), TestFieldExistsQuery (30) and TestIndexOrDocValuesQuery (5, 1 skipped) pass; all of org.apache.lucene.search.* and org.apache.lucene.document.* with the repro: 275 classes, 2,849 tests, 0 failures, 75 skipped.
• Earlier checks: the composite decomposition harnesses go from 600 of 17,366 and 69 of 2,534 mismatching query/composite checks to 0, the factory reach probe from 54 silently wrong cases to 0 (with the fix its summary reads ok=63 bad=8: all 8 are loud IllegalStateExceptions that 10.2.2 throws too), and a Z3 model of the rewrite decision (3 leaves x 2 documents, 11 queries) finds a counterexample with the shipped condition on MultiReaders and none with the fix or on a single strict index.

Keep the MatchAllDocsQuery and MatchNoDocsQuery rewrites, which use reader-wide counts and are sound on composites, and fall back to super.rewrite(searcher) where the FieldExistsQuery branch is. The weight's per-leaf shortcuts stay (for example count returns a leaf's point doc count when its own bounds are inside the range, PointRangeQuery.java:353-357).

Trade-offs

Simplest to reason about, but it also gives up the FieldExistsQuery rewrite on consistent indexes, which TestPointQueries.testRangeQueryOptimizesRewrites asserts and #14609 wanted ("rewriting at the shard level enables other optimizations downstream").

Method: non-formal (the first signal came from code reading and an executed probe; the Z3 model came later).

• Lead. A review of features new in 10.x read the #14609 rewrite with the property "on every composite reader a point range query matches exactly the documents that have a point in range, leaf by leaf".
• Probe. Public-API probes on the stock main jar: 6 hits instead of 3 in a larger probe and a count of 2 instead of 1 in the minimal one, on main, 10.6.0 and 10.3.0; 10.2.2 correct.
• Generalisation. A composite-reader count-decomposition oracle (a MultiReader count must equal the sum of the per-index counts) found that every mixed-schema mismatch among the layouts it tried has this root.
• Reduction and model. One JUnit test; a Z3 model of the rewrite decision (3 leaves x 2 documents, 11 queries) finds a counterexample with the shipped condition on MultiReaders, none with the fix or on a single strict index, and shows that positive ranges can only over-match (lost hits need MUST_NOT).