MemoryIndex reports the total number of point dimensions as the number of index dimensions

FieldType.setDimensions(dimensionCount, indexDimensionCount, bytes) lets a point field store more dimensions than it indexes (selective indexing, LUCENE-8496, 7.6.0). PointValues reports the two counts separately, and PointRangeQuery, PointInSetQuery and the sandbox MultiRangeQuery require a query to have exactly getNumIndexDimensions() dimensions. MemoryIndex stores both counts correctly in its FieldInfo, but its PointValues returns the total count from getNumIndexDimensions() too (MemoryIndex.java:2194-2197, a copy of getNumDimensions() at :2189-2192). So a MemoryIndex holding the point (10, 5) of a 2-dimension, 1-index-dimension field rejects IntPoint.newExactQuery("f", 10) with IllegalArgumentException, while an IndexWriter index of the same document answers it. Monitor, which matches a single document through a MemoryIndex, reports the stored query as an error and no match. Every release from 7.6.0 to 10.6.0 and main is affected; built-in types are not, because they index all their dimensions (or, for shapes, are queried by shape queries that never read the count). The fix is one line, but on its own it turns some set queries, on documents that hold several values in the field, from an exception into a silent miss because of the separate PointInSetQuery bug LFV-006, so LFV-006's query-side fix should land first or together with it.

• Data dimensions (getNumDimensions(), FieldInfo.getPointDimensionCount()): how many numbers each point value stores.
• Index dimensions (getNumIndexDimensions(), FieldInfo.getPointIndexDimensionCount()): how many of them, from the first, the index is built on, and how many dimensions a point query on the field must have.
• Selectively indexed field: a field with fewer index dimensions than data dimensions, made with FieldType.setDimensions(n, m < n, bytes) (FieldType.java:309-310, "Enables points indexing with selectable dimension indexing"). The two-argument setDimensions(n, bytes) that the other built-in point types use passes n for both (:305-306). The only built-in selectively indexed type is ShapeField (7 dimensions, 4 indexed, ShapeField.java:52).

1. MemoryIndex stores both counts in the field's FieldInfo (createFieldInfo, MemoryIndex.java:775-776; setPointDimensions, :745-750; the FieldInfo rebuilt for doc values keeps both, :904-905).
2. Its PointValues returns the data count from both methods: getNumDimensions() at :2189-2192 and getNumIndexDimensions() at :2194-2197 both return info.fieldInfo.getPointDimensionCount();.
3. PointRangeQuery.rewrite checks every leaf (PointRangeQuery.java:544-549) and throws unless getNumIndexDimensions() equals the query's dimension count (:621-633). PointInSetQuery (PointInSetQuery.java:168-174) and the sandbox MultiRangeQuery (MultiRangeQuery.java:299-306) have the same check. A query built for the field's real index count is rejected.
4. Shape queries never read getNumIndexDimensions(), so ShapeField in a MemoryIndex is not affected by this check. (Its reported count is wrong too on stock code, 7 instead of 4, but a LatLonShape box query gives the same hit and miss with and without the fix; executed.)

Field type setDimensions(2, 1, Integer.BYTES), one point (10, 5), query IntPoint.newExactQuery("f", 10). MemoryIndex: FieldInfo.getPointIndexDimensionCount()=1, PointValues.getNumDimensions()=2, PointValues.getNumIndexDimensions()=2, and the query throws. IndexWriter index of the same document: getNumDimensions()=2, getNumIndexDimensions()=1, count 1.

• Who is hit. Code that puts a custom selectively indexed point field into a MemoryIndex: MemoryIndex.addField, MemoryIndex.fromDocument (MemoryIndex.java:531-564), or Monitor, whose single-document batch is a MemoryIndex (DocumentBatch.java:101-112). Only one-document batches are hit: a batch of several documents is built with an IndexWriter over a ByteBuffersDirectory (DocumentBatch.java:55-62, :66-92), which reports the right count. Typical uses are matching one incoming document against many stored queries and testing a document against a query before indexing it.
• Opt-in. The other built-in point types index all their dimensions; ShapeField is selectively indexed but its queries never read the count. Such a custom type is legal and works with IndexWriter.
• What happens. Every PointRangeQuery (exact or range), PointInSetQuery and sandbox MultiRangeQuery with the field's real index dimension count throws IllegalArgumentException, deterministically, with assertions on and off. One point is enough.
• Through Monitor it becomes a missing match. Monitor catches the exception and reports it per query (Monitor.java:294-302). Executed: a stored IntPoint.newExactQuery("f", 10) and a document holding (10, 5) give matchCount=0 errors={q1=java.lang.IllegalArgumentException: field="f" was indexed with numIndexDimensions=2 but this query has numDims=1}, with assertions on and off; with the fix matchCount=1. An application that does not read getErrors() sees no match.
• The reverse (executed): a query with as many dimensions as the field stores passes the check on a MemoryIndex and is evaluated against the full stored value: IntPoint.newRangeQuery("f", new int[] {10, 5}, new int[] {10, 5}) matches and {10, 6} does not, while an IndexWriter index rejects both (numIndexDimensions=1 but this query has numDims=2). Not behaviour to rely on: with the fix, a MemoryIndex rejects them too.
• How likely. Rare: no prior public report found since 7.6.0 (2018).

The test sits in org.apache.lucene.index.memory next to the module's tests but uses no package-private API. It uses no randomness, and no seed sensitivity was observed.

As the regression test: the same test, for example as a new method of TestMemoryIndex; the try/catch can go (assertTrue(index.search(IntPoint.newExactQuery("f", 10)) > 0);). It fails on stock main and passes with the fix, with assertions on and off (gradle); :lucene:memory:check -x test passes with the fix and the test in place.

Not shown by this test, executed elsewhere: set queries (on stock code they throw an IllegalArgumentException too, with PointInSetQuery's own wording was indexed with numIndexDims=2 but this query has numIndexDims=1), Monitor (an error entry and no match), and the release jars 7.6.0 to 10.6.0.

cp TestMemoryIndexSelectiveIndexing.java $LUCENE/lucene/memory/src/test/org/apache/lucene/index/memory/
cd $LUCENE && ./gradlew :lucene:memory:test --tests TestMemoryIndexSelectiveIndexing -Ptests.seed=DEADBEEF -Ptests.asserts=true
# assertions off: -Ptests.asserts=false instead (same result)

# gradle, main 645d4fe14a20, -Ptests.seed=DEADBEEF -Ptests.asserts=true (the same with =false); excerpt: the SeedInfo frame and the org.apache.lucene.core@11.0.0-SNAPSHOT/ module prefixes are left out
TestMemoryIndexSelectiveIndexing > testExactQueryOnOneOfTwoDimensions FAILED
    java.lang.AssertionError: expected f:10 to match, but got java.lang.IllegalArgumentException: field="f" was indexed with numIndexDimensions=2 but this query has numDims=1
        at org.apache.lucene.index.memory.TestMemoryIndexSelectiveIndexing.testExactQueryOnOneOfTwoDimensions(TestMemoryIndexSelectiveIndexing.java:35)

        Caused by:
        java.lang.IllegalArgumentException: field="f" was indexed with numIndexDimensions=2 but this query has numDims=1
            at org.apache.lucene.search.PointRangeQuery.checkValidPointValues(PointRangeQuery.java:632)
            at org.apache.lucene.search.PointRangeQuery.rewrite(PointRangeQuery.java:548)
            at org.apache.lucene.search.IndexSearcher.rewrite(IndexSearcher.java:873)
            at org.apache.lucene.search.IndexSearcher.rewrite(IndexSearcher.java:884)
            at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:753)
            at org.apache.lucene.index.memory.MemoryIndex.search(MemoryIndex.java:1115)
            at org.apache.lucene.index.memory.TestMemoryIndexSelectiveIndexing.testExactQueryOnOneOfTwoDimensions(TestMemoryIndexSelectiveIndexing.java:32)

# releases 7.6.0 to 10.6.0: the same message; the check sits in the weight's scorerSupplier up to 10.2.2 and in rewrite from 10.3.0

• Contract. PointValues.java:483-487: getNumDimensions() "Returns how many dimensions are represented in the values", getNumIndexDimensions() "Returns how many dimensions are used for the index". FieldType.setDimensions(int, int, int): "Enables points indexing with selectable dimension indexing" (FieldType.java:309-310); (2, 1, 4) passes its validation (:323-329).
• MemoryIndex promises arbitrary queries. createSearcher(): "a searcher that can be used to execute arbitrary Lucene queries" (MemoryIndex.java:1074); search(Query): "an arbitrary Lucene query to run against this index" (:1104). No javadoc, CHANGES or MIGRATE entry excludes selectively indexed points.
• Parity. IndexWriter reports 1 index dimension and count 1 for the same input; the independent reviewer's 3-data / 2-index LongPoint control on IndexWriter matches too.
• History. LUCENE-8496 (1118299c338, 2018-10-08) split MemoryIndex's getNumDimensions() into getNumDataDimensions() and a new getNumIndexDimensions() with the same body, in the commit that also passed pointIndexDimensionCount() into MemoryIndex's FieldInfo and made the point queries check getNumIndexDimensions(). Two later edits (LUCENE-9149 rename, 2020; #16387, 2026) kept the value.
• No test. lucene/memory/src/test only calls setDimensions(1, 4) (TestMemoryIndex.java:673, :699).

The independent reviewer's strongest case: PointValues, PointRangeQuery and PointInSetQuery are @lucene.experimental, MemoryIndex has a simplified single-document point tree, and its tests never use unequal counts, so selective indexing might have been meant only for BKD-backed readers.

Why it fails: no such limitation is documented; MemoryIndex promises arbitrary queries; the introducing commit deliberately stored the index count in MemoryIndex's FieldInfo; IndexWriter handles the same input; and the two methods have different documented meanings, so returning the same value from both cannot be the design. The reviewer rated the argument weak. Verdict: a defect.

Both A and A with LFV-006's query-side fix were run through gradle in a scratch worktree of 645d4fe14a20. The MemoryIndex line is the whole change for this issue; the coupling is the order of landing.

One line in MemoryIndexPointValues.getNumIndexDimensions() (MemoryIndex.java:2196). Land it after, or together with, LFV-006's query-side fix in PointInSetQuery (merge visitor only when numDims == 1 && values.getNumDimensions() == 1; SinglePointVisitor compares only the index-dimension prefix).

       @Override
       public int getNumIndexDimensions() {
-        return info.fieldInfo.getPointDimensionCount();
+        return info.fieldInfo.getPointIndexDimensionCount();
       }

Trade-offs

• The MemoryIndex change is safe for every built-in type: the other built-in point types index all their dimensions, so their index count equals their data count; ShapeField is selectively indexed (7 dimensions, 4 indexed), but shape queries never read the count: a LatLonShape box query in a MemoryIndex gives the same results with and without the fix (executed).
• Why the pairing: with the one line alone, some set queries, on documents that hold several values in the field, go from an exception to a silent miss; single-valued documents, and values added in increasing order of the indexed dimension, still match (executed). 2 dimensions / 1 indexed, values (30, 5) then (10, 5), IntPoint.newSetQuery("f", 10): no match, with assertions on and off. MemoryIndex sorts values only when the field has 1 dimension in total (MemoryIndex.java:1332-1338), and PointInSetQuery's 1-dimension merge visitor needs increasing indexed values (PointInSetQuery.java:203, :282). 3 dimensions / 2 indexed, values (1, 2, 100) and (3, 4, 300), a 2-dimension set query for (1, 2): no match with assertions off, and Lucene's own assert packedValue.length == pointBytes.length (PointInSetQuery.java:414) with assertions on. Both are LFV-006's root cause, reached through MemoryIndex.
• The tested exact and range queries are correct with the one line alone: the repro, 1-dimension exact and range queries on the 2/1 field (through addField and fromDocument), a sandbox MultiRangeQuery, and the independent reviewer's 3/2 range query. PointRangeQuery compares only the first numDims dimensions of a value or cell (PointRangeQuery.java:134-147, :594-618).
• Behaviour change: with the fix, a query with as many dimensions as the field stores (2 on the 2/1 field) throws on a MemoryIndex, as it already does on an IndexWriter index; stock code matched it against the full stored value (executed).

What the checks validated

• Gradle in a scratch worktree of 645d4fe14a20 (2026-11-01): the repro fails on stock main and passes with the one line, each with -Ptests.asserts=true and =false.
• With the one line: :lucene:memory:test 4 classes with the repro, 52 tests, 0 failures; :lucene:monitor:test 32 classes, 181 tests, 0 failures, 1 skipped; :lucene:memory:check -x test (google-java-format, ecj lint, forbidden APIs) passes with the test in place.
• With the one line plus LFV-006's query-side fix: the repro and both set-query checks pass with assertions on and off; :lucene:memory:test 5 classes, 54 tests, 0 failures; core TestPointQueries 56 tests, 0 failures, 2 skipped.
• Earlier, the same line: the repro with the fix shadowed in (assertions on and off); the independent reviewer's 4-test class 4/0 and upstream TestMemoryIndex 30/0; the Monitor check matchCount=1.

The same diff, landed before LFV-006 is fixed.

Trade-offs

Fixes exact and range queries, but turns some set queries, on documents that hold several values in the field, from a loud exception into a silent miss (executed above). Not recommended unless LFV-006's fix lands in the same release.

Also sort point values when the field has one index dimension (MemoryIndex.java:1334 tests the total count today), keeping the per-dimension min/max loop.

Trade-offs

Would make 1-index-dimension set queries in a MemoryIndex correct without the core change, but not multi-index-dimension set queries (SinglePointVisitor), and nothing for on-disk indexes, so LFV-006 is still needed. Only worth it if LFV-006's fix is delayed. Not built.

Method: non-formal (a side observation from code reading). In the 2026-10-22 study, the verifier of the selective-indexing PointInSetQuery bug (LFV-006) read MemoryIndex because it sorts points only for 1-dimensional fields, and noted that getNumIndexDimensions() returns getPointDimensionCount(). A follow-up executed it with its own tests (testS2MemoryIndexExact, testS2MemoryIndexSet) and a fix matrix that showed that with the fix alone its set query, on a document holding (30, 5) then (10, 5), becomes a silent miss.

The hunt then cut it to the one-test repro, re-ran it on main 645d4fe14a20 and on 10.5.2, 10.6.0 and 9.12.3 with controls, had an independent reviewer from another model family (OpenAI GPT-5.6) try to refute it with its own 3-data / 2-index-dimension test (verdict real, high confidence), and ran the history sweep over 7.5.0-10.6.0.