Shape doc-values box queries stay cacheable after a doc-values update, so the query cache returns the old matches

The finding in 30 seconds

The query cache remembers, per segment, which documents a filter matched. After an in-place doc-values update the segment keeps its cache identity, so every query that reads doc values must tell the cache not to reuse the old answer. The shape doc-values box query never does.

Summary card. Step 1: s1.count(q) with q = LatLonShape.newSlowDocValuesBoxQuery("shape", INTERSECTS, 0, 10, 0, 10) returns 3 (docs 0, 1, 2) and the cache stores {0, 1, 2} under (q, core of segment _0). Step 2: w.updateDocValues moves doc 0 to (2, 13), outside the box; r2 = openIfChanged(r1, w) keeps the same segment core. Step 3: s2.count(q): isCacheable is still true (SpatialQuery.queryIsCacheable returns true), so a cache hit hands back {0, 1, 2}. Expected 2 (docs 1, 2); actual 3 (docs 0, 1, 2), doc 0 outside yet counted. A map shows the box and the six points, with doc 0's move. Notes: a document moved into the box is not found (3, not 4); FILTER and MUST_NOT clauses use the stale set the same way; it lasts across later reopens until a merge writes a new segment or the cache evicts the entry; controls (no cache, a fresh cache, the binary doc-values range query, and the fix) are right. Footer: affected 9.4.0 to 10.5.2 and main; on 9.4.0-10.5.2 a plain new IndexSearcher shares the default LRUQueryCache, on main only an installed cache does. — The diagram's own small example: six point shapes and a box over latitude and longitude 0-10 that holds docs 0-2. The cache stores {0, 1, 2}; `updateDocValues` moves doc 0 out, the reopened segment keeps its core, and the cached count stays 3 instead of 2. Select the still to open it at full size. Generated with AI, checked against the code.

Index 20 documents with a doc-values-only LatLonShape field (createDocValueField): doc i is a small square at (i, i). newSlowDocValuesBoxQuery over latitude and longitude -1 to 4.5 matches docs 0-4: 5 hits, and the query cache stores that answer for the segment.
Move doc 0's square to (60, 60) in place with IndexWriter#updateDocValues and reopen. You expect 4 hits, which is what a searcher without a cache returns.
Instead the cached search returns 5: doc 0 still matches. The same with XYShape, on every release from 9.4.0 to 10.5.2 and on main (assertions off also checked on 9.4.0, 10.0.0, 10.5.2 and main).
Why: the query reads each shape from doc values, but its isCacheable comes from SpatialQuery and is always true. The cache keys its entries on the segment core, which a doc-values update does not change, so it reuses the old answer.
Only uses that do not need scores read the cache (a filter or exclusion clause, ConstantScoreQuery, count()); a relevance-scored search is correct. The stale answer lasts until the segment is merged into a new one, the entry is evicted or the cache is cleared. On 9.4.0-10.5.2 the default IndexSearcher cache is enough (12,000-document segment: 5 instead of 4, run on 9.4.0, 9.12.3, 10.0.0 and 10.5.2); main has no default cache, so it needs an installed one.

Who is affected

Applications with a doc-values-only LatLonShape or XYShape field that move shapes with updateDocValues and repeat newSlowDocValuesBoxQuery as a filter, exclusion or count across reopens. On 9.4.0-10.5.2 the default searcher cache is enough for large segments (10,000+ documents and a big share of the index); main needs an installed cache. Indexed shape fields cannot be updated, so they are safe.

Mechanism, terms and numbers596 words

LatLonShape.newSlowDocValuesBoxQuery and XYShape.newSlowDocValuesBoxQuery build subclasses of BaseShapeDocValuesQuery extends SpatialQuery, which read every candidate's shape from BinaryDocValues (BaseShapeDocValuesQuery.java:64). Their weight's isCacheable is SpatialQuery's, which returns queryIsCacheable(ctx) (SpatialQuery.java:221-222), a constant true (:146-148) that no class overrides. SegmentCacheable requires doc-values readers to return DocValues#isCacheable, because "Updated doc values fields are not suitable for caching" (SegmentCacheable.java:30-32), and LRUQueryCache trusts isCacheable and keys entries on the segment's core cache helper (LRUQueryCache.java:816-834), which by design ignores doc-values updates (LeafReader.java:65-70). So after IndexWriter#updateDocValues moves a document's shape in a doc-values-only field and the reader is reopened, a cache entry built before the update keeps being served to every use that does not need scores (filter and exclusion clauses, ConstantScoreQuery, count()): in the repro 5 hits instead of 4, the moved document still matching, until the segment is merged into a new one or the entry is evicted or cleared. Relevance-scored searches skip the cache and are correct. All 29 releases from 9.4.0 to 10.5.2 fail the repro with an installed cache, and on these releases the default IndexSearcher cache is enough (zero configuration run on 9.4.0, 9.12.3, 10.0.0 and 10.5.2); main needs an installed cache (no default since GITHUB#14187). The introducing PR (#1017) had a DocValues.isCacheable check in its first commit, which an in-PR refactor dropped. A one-method override fixes it.

Terms

Query cache: LRUQueryCache stores, per segment, the documents a filter clause matched (and their count), keyed on the query and the segment's core cache key (LRUQueryCache.java:827-834). The searcher puts it in front of a query only where scores are not needed (IndexSearcher.java:979-981): filter and exclusion clauses, ConstantScoreQuery, count(). Before it looks anything up, it asks the weight whether the segment may be cached: Weight#isCacheable(ctx) (:816, and :903 for count()).
Core cache key: identifies a segment's immutable core. "Two readers that have the same data but different sets of deleted documents or doc values updates may be considered equal" (LeafReader.java:65-70). After updateDocValues and a reopen, the segment keeps its key.
The rule: "Objects that depend on doc values should return DocValues#isCacheable(LeafReaderContext, String...), which will check to see if the doc values fields have been updated. Updated doc values fields are not suitable for caching" (SegmentCacheable.java:30-32). DocValues.isCacheable returns false once the field has a doc-values update generation (DocValues.java:340-346).
Shape doc values: LatLonShape.createDocValueField and XYShape.createDocValueField store a shape as BINARY doc values only, "without creating indexable fields" (LatLonShape.java:89-92; ShapeDocValuesField.java:46-52, @lucene.experimental at :38). IndexWriter#updateDocValues accepts BINARY updates on such a field (IndexWriter.java:2081-2127).

The bug

newSlowDocValuesBoxQuery (LatLonShape.java:262-284, XYShape.java:211-217) builds a LatLonShapeDocValuesQuery or XYShapeDocValuesQuery, both final subclasses of BaseShapeDocValuesQuery extends SpatialQuery (BaseShapeDocValuesQuery.java:40).
The scorer reads reader.getBinaryDocValues(field) (:64) and decodes each candidate's shape in matches() (:77-79). The class overrides only getScorerSupplier.
The weight is SpatialQuery's ConstantScoreWeight: isCacheable(ctx) returns queryIsCacheable(ctx) (SpatialQuery.java:221-222), which returns true (:146-148). Nothing in lucene/ overrides it.
After the update and a reopen, isCacheable still says true for the updated segment, the core cache key is unchanged, and LRUQueryCache returns the entry computed from the old shapes: the old doc-id set, or for count() the old count (LRUQueryCache.java:914-924).
Deletions are not affected: live docs are applied on top of a cached set, and count() skips the cache on a segment with deletions (:893-896); a later delete was honoured in a control run.

Exposure and likelihood564 words

Who is exposed. Applications that keep shapes in a doc-values-only LatLonShape or XYShape field, move them in place (IndexWriter#updateDocValues or updateBinaryDocValue; for example moving geofences or re-geocoded assets) and repeat the same newSlowDocValuesBoxQuery as a filter, an exclusion clause or a count (any use that does not need scores) across reopens.
What it takes. (a) A doc-values-only field: a field name that also has indexed shape triangles cannot be updated in place (verifyOrCreateDvOnlyField, IndexWriter.java:2097; points rejected at FieldInfos.java:619-626), so the "indexed + doc values" pairing is not affected. (b) An in-place update. (c) The same query cached before the update. (d) A query cache that survives the reopen.
Default vs opt-in. On 9.4.0-10.5.2 zero configuration is enough: the default IndexSearcher cache is a JVM-wide LRUQueryCache (1,000 queries, at most 32 MB; 10.5.2 IndexSearcher.java:96-99) that caches segments of at least 10,000 documents holding more than half the average segment size (9.6.0 onward; 10.5.2 LRUQueryCache.java:175-198; on 9.4.0-9.5.0 at least 10,000 documents and at least 3% of the index, 9.4.0 LRUQueryCache.java:150, :170-171), and the default policy caches this query after 5 uses (10.5.2 UsageTrackingQueryCachingPolicy.java:122-137). A zero-configuration program with a 12,000-document segment (accepted by both size rules) prints 5 instead of 4 on 9.4.0, 9.12.3, 10.0.0 and 10.5.2; it was not run on the other 25 releases. branch_10_6 (10.6.0, released 2026-10-29) and the unreleased branch_10x still have that default cache (read only). On main "Caching is disabled by default" (IndexSearcher.java:82-84, GITHUB#14187, unreleased), so it needs setQueryCache or setDefaultQueryCache. An installed new LRUQueryCache(maxSize, maxRamBytesUsed) applies the same 10,000-document rule (LRUQueryCache.java:252, :277-296) and needs RAM for five dense bit sets over the index (:792-808); only a custom leaf predicate, as in the repro, accepts any segment size.
What you see. Silent wrong match decisions for the moved documents of that segment, in both directions: a document moved out of the box still matches, and one moved into it is missed (in a separate executed example, 3 hits instead of 4). Every use of the box query that does not need scores sees it: a FILTER or MUST_NOT clause, ConstantScoreQuery, count() of the query or of a query that contains it, and (by reading the code) a search sorted by a field without scores, including a later request that never ran before. A relevance-scored use (the box as the top-level query of search(q, n), or a scored MUST/SHOULD clause) does not use the cache and is correct: a fact-check probe on main returned [1, 2, 3, 4] for those both ways, while ConstantScoreQuery, FILTER, MUST_NOT and both counts were stale. No error.
How long. Until the segment is merged into a new segment (forceMerge cleared it), the LRU evicts the entry, or the application clears the cache (clear(), clearQuery(q)) or replaces it (eviction and clearing read from the code, not run); it survives further reopens. Each updated segment serves only its own old decisions: no document of another segment leaks in, and an update that moves documents in several segments makes each of them stale.
How likely. Rare: shape doc values are @lucene.experimental, the query is the slow doc-values variant, and a code search found no Elasticsearch, OpenSearch or Solr code that runs it or updates shape doc values in place.

Minimal repro

The test shows that LatLonShape.newSlowDocValuesBoxQuery stays cacheable after an in-place doc-values update, so LRUQueryCache keeps serving the result from before the update.

TestShapeDocValuesQueryCachedAfterUpdate.java (JUnit test)71 lines

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package org.apache.lucene.document;

import org.apache.lucene.document.ShapeField.QueryRelation;
import org.apache.lucene.index.DirectoryReader;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.index.Term;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.LRUQueryCache;
import org.apache.lucene.search.Query;
import org.apache.lucene.search.QueryCachingPolicy;
import org.apache.lucene.store.ByteBuffersDirectory;
import org.apache.lucene.store.Directory;
import org.apache.lucene.tests.util.LuceneTestCase;

public class TestShapeDocValuesQueryCachedAfterUpdate extends LuceneTestCase {

  private static final QueryCachingPolicy ALWAYS_CACHE =
      new QueryCachingPolicy() {
        @Override
        public void onUse(Query query) {}

        @Override
        public boolean shouldCache(Query query) {
          return true;
        }
      };

  public void testCachedCountSeesDocValuesUpdate() throws Exception {
    try (Directory dir = new ByteBuffersDirectory();
        IndexWriter w = new IndexWriter(dir, new IndexWriterConfig())) {
      Document doc = new Document();
      doc.add(new StringField("id", "0", Field.Store.NO));
      doc.add(LatLonShape.createDocValueField("shape", 0, 0));
      w.addDocument(doc);
      Query query =
          LatLonShape.newSlowDocValuesBoxQuery("shape", QueryRelation.INTERSECTS, -1, 1, -1, 1);
      LRUQueryCache cache = new LRUQueryCache(1000, 1 << 20, ctx -> true, Float.MAX_VALUE);
      try (DirectoryReader r1 = DirectoryReader.open(w)) {
        IndexSearcher s1 = new IndexSearcher(r1);
        s1.setQueryCache(cache);
        s1.setQueryCachingPolicy(ALWAYS_CACHE);
        assertEquals(1, s1.count(query));
        w.updateDocValues(new Term("id", "0"), LatLonShape.createDocValueField("shape", 10, 10));
        try (DirectoryReader r2 = DirectoryReader.openIfChanged(r1, w)) {
          IndexSearcher s2 = new IndexSearcher(r2);
          s2.setQueryCache(cache);
          s2.setQueryCachingPolicy(ALWAYS_CACHE);
          // expected 0 (the only shape was moved out of the box); actual 1, served from the cache
          assertEquals("count after the doc-values update", 0, s2.count(query));
        }
      }
    }
  }
}

Command

# in a copy of this finding's repro folder (repro/LFV-308/, which holds src/)
# Lucene checkout at the tested commit, built with: ./gradlew --no-daemon assemble   (JDK 25)
LUCENE=/path/to/lucene; V=12.0.0-SNAPSHOT
CP=$LUCENE/lucene/core/build/libs/lucene-core-$V.jar
CP=$CP:$LUCENE/lucene/codecs/build/libs/lucene-codecs-$V.jar
CP=$CP:$LUCENE/lucene/test-framework/build/libs/lucene-test-framework-$V.jar
CP=$CP:randomizedtesting-runner-2.9.2.jar:junit-4.13.2.jar:hamcrest-3.0.jar   # test-framework dependencies (Maven Central)
CP=$CP:junit-platform-console-standalone-6.1.3.jar   # JUnit Platform console launcher (Maven Central)

javac -d classes -cp "$CP" $(find src -name '*.java')

# run 1: assertions on, seed DEADBEEF
java -ea -esa --add-modules jdk.incubator.vector --enable-native-access=ALL-UNNAMED -Dtests.seed=DEADBEEF \
     -cp "classes:$CP" org.junit.platform.console.ConsoleLauncher \
     execute --select-class org.apache.lucene.document.TestShapeDocValuesQueryCachedAfterUpdate

Expected vs actual: Expected count 0 after the only shape is moved out of the box and the reader is reopened; actual 1, from the cache (expected:<0> but was:<1>), with -ea and -da.

All files of this repro: repro/LFV-308/

More files, environment, full output38 words

Environment

lucene	`Lucene main e1faeba08bde (2026-11-02)`
jdk	`25.0.3+9-LTS`

Observed output

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: count after the doc-values update expected:<0> but was:<1>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:647)
       org.apache.lucene.document.TestShapeDocValuesQueryCachedAfterUpdate.testCachedCountSeesDocValuesUpdate(TestShapeDocValuesQueryCachedAfterUpdate.java:66)

Why it looks unintended

SegmentCacheable: "Objects that depend on doc values should return DocValues#isCacheable ... Updated doc values fields are not suitable for caching."
Sibling doc-values queries return DocValues.isCacheable(ctx, field), and TestLRUQueryCache asserts that an updated segment is not served from the cache.
The introducing PR's first commit had this check; an in-PR refactor dropped it, with no review comment about caching.

The full argument243 words

The documented rule. SegmentCacheable.java:30-32 (quoted above). It says "should": strong guidance that the cache relies on, not something it checks. QueryCache.java:27-30: the cached weight "will only be equivalent if scores are not needed"; here it is not equivalent even with scores off.
Upstream tests assert the rule for other queries. TestLRUQueryCache#testDocValuesUpdatesDontBreakCache (TestLRUQueryCache.java:2222) shares one cache across reopens and asserts that after updateNumericDocValue the segment is "no longer cached due to DV update"; TestSegmentCacheables#testMultipleDocValuesDelegates builds SegmentCacheable lambdas over DocValues.isCacheable and asserts that, after field3 is updated, every composite that refers to it is uncacheable (TestSegmentCacheables.java:43-92).
The siblings follow it. LatLonDocValuesBoxQuery.java:164-165, LatLonDocValuesQuery.java:169-170, XYDocValuesPointInGeometryQuery.java:141-142 and BinaryRangeFieldRangeQuery.java:146-147 return DocValues.isCacheable(ctx, field). So does Elasticsearch's own, independent shape doc-values query.
History: the check existed and was lost. In PR #1017's first commit (3554306677, 2022-05-18) the doc-values query class had isCacheable(ctx) { return DocValues.isCacheable(ctx, field); }. Commit 2cf24b47c0b6 ("Major refactor to leveral SpatialQuery" [sic], 2022-07-20) replaced that class with BaseShapeDocValuesQuery extends SpatialQuery, added queryIsCacheable returning true and routed Weight#isCacheable through it, without an override. None of the PR's 10 comments or 30 review comments mentions caching; no javadoc, CHANGES or MIGRATE entry limits updates or caching for shape doc values.
The trigger is legal. Public factories, a documented doc-values-only field, and an update path that accepts BINARY updates on doc-values-only fields (IndexWriter.java:2081-2127). The upstream DV query tests (TestLatLonPolygonShapeDVQueries and siblings) index only the doc-values field.

The devil's-advocate check157 words

The intended-behaviour checker built the strongest case: shape doc values and LRUQueryCache are @lucene.experimental, SpatialQuery is @lucene.internal (SpatialQuery.java:58), the shape doc-values field was designed as a companion of the indexed triangles (LUCENE-10654 motivates it with IndexOrDocValuesQuery), and main does not cache by default, so this is an unsupported combination.

Why it fails: @lucene.experimental is about API stability, not permission to return wrong results. The doc-values-only field is documented (LatLonShape.java:89) and is what the upstream DV tests use, and a companion field of the same name cannot be updated at all, so the only updatable case is exactly the doc-values-only one. SpatialQuery being internal does not matter: users reach it through public factories. The missing default cache on main narrows reach but does not change the rule, and 9.4.0-10.5.2 reproduce with zero configuration. The first PR commit had the right check. No verifier or reviewer rated the intended argument better than weak.

Suggested fix

Recommended Override queryIsCacheable in BaseShapeDocValuesQuery (validated)

Override queryIsCacheable in BaseShapeDocValuesQuery to return DocValues.isCacheable(ctx, field), as the sibling doc-values queries do (+8 lines, no public API change). The test then passes 3 of 3 with assertions on and off, and 63 suites (547 tests) of the document package and the query-cache tests pass.

@Override
protected boolean queryIsCacheable(LeafReaderContext ctx) {
  // matches are read from doc values, which may be updated in place
  return DocValues.isCacheable(ctx, field);
}

All options and trade-offs3 options · 450 words

One fix was built and validated. Two alternatives that move the rule into SpatialQuery were not built.

Override queryIsCacheable in BaseShapeDocValuesQuery (validated) (recommended)

In BaseShapeDocValuesQuery, add the override below and import org.apache.lucene.index.DocValues and org.apache.lucene.index.LeafReaderContext. 1 file, +8 lines. SpatialQuery and BaseShapeDocValuesQuery are package-private, so there is no public API change.

@Override
protected boolean queryIsCacheable(LeafReaderContext ctx) {
  // matches are read from doc values, which may be updated in place
  return DocValues.isCacheable(ctx, field);
}

Trade-offs

Segments whose shape doc values were updated are no longer cached until they are merged, the same behaviour as the four sibling doc-values queries named under Why it is not intended; one FieldInfo lookup per segment. It fixes only this subclass family; a future doc-values subclass of SpatialQuery would have to remember the override. No benchmark was run.

What the checks validated

Scratch worktree of a9779ef3c193, gradle, JDK 25: stock + the test fails 2 of 3 with assertions on (seed DEADBEEF) and off (seed 5EED0308); + the fix passes 3 of 3 with both.
With the fix, :lucene:core:test over the whole org.apache.lucene.document package plus TestLRUQueryCache, TestSegmentCacheables and TestUsageTrackingFilterCachingPolicy (seed DEADBEEF, assertions on): 63 suites, 547 tests, 29 skipped, 0 failures, including the 7 upstream shape doc-values suites and the new test (run with the test's first text, which differs only in an unused lambda parameter's name); in 6 of those upstream suites testRandomBig was skipped (nightly only).
:lucene:core:checkGoogleJavaFormat, ecjLintMain, ecjLintTest, forbiddenApisMain and forbiddenApisTest pass with the fix and the test.
10.5.2 jars with the fixed class shadowed in: the JUnit passes 3 of 3 (assertions off) and the zero-configuration main() prints OK.
Earlier, on main ea57360912ed: the same change passed the finder's and the independent reproducer's tests and the FILTER/MUST_NOT probe (the probe's only fix check); a 9.x variant of the file passed on 9.4.0 and 9.12.3, and the change on 10.0.0 and 10.5.2.
Not run: the full lucene/core suite.

Make SpatialQuery's default check doc-values updates (not built)

Change SpatialQuery.queryIsCacheable (SpatialQuery.java:146-148) to return DocValues.isCacheable(ctx, field);.

Trade-offs

Protects any future doc-values subclass. For the BKD-backed subclasses (LatLonShapeQuery, LatLonShapeBoundingBoxQuery, LatLonPointQuery, XYShapeQuery) the answer stays true: a field also indexed with points cannot be updated in place (FieldInfos.java:619-626), so its doc-values generation stays -1 (DocValues.java:343). The cost is that the base class of the point and triangle queries then reasons about doc values, plus a cheap FieldInfo lookup per segment for every spatial query.

Make queryIsCacheable abstract (not built)

Remove the default so that every SpatialQuery subclass family states its own rule.

Trade-offs

The most explicit, with a larger diff over all subclasses; still internal API only.

Downstream note

If an application uses this query as a region restriction, moved documents keep their old decision in non-scoring uses until the segment is merged or the entry leaves the cache: one moved out of an allowed region is still returned, one moved into an excluded region is still shown; deletes are still honoured.

How it was found

A scan of the bytecode of every isCacheable implementation in Lucene flagged SpatialQuery's constant true in a class family that reads binary doc values. A sweep of the real jars that compares cached and uncached results after an in-place update (35 doc-values query cases) returned stale results for both shape queries; a minimal JUnit confirmed it.

The method in full186 words

Method: semi-formal (the hunt's middle tier): a bytecode scan and a differential sweep, with a TLA+ model written right after.

The hunt unit checked query-cache correctness: every Weight#isCacheable must be false where the cached answer could differ from a fresh one.
A scan with the JDK ClassFile API listed every isCacheable implementation in the main jars (123 classes) and what it returns. It flagged SpatialQuery's weight, which delegates to queryIsCacheable, a constant true never overridden, in a class family whose doc-values subclass reads BinaryDocValues.
A differential sweep on the real jars ran 35 doc-values query cases: fill a cache, update doc values in place, reopen with the same segment core, compare the cached with the uncached result. Both shape queries returned STALE (27 ok and 8 STALE cases in all, the STALE ones explained by three separate root causes).
A TLA+ model of the LRUQueryCache path (QueryCacheDV.tla) violates its SearchCorrect invariant with a 5-state counterexample for a constant-true isCacheable and holds for the correct shapes.
The finder reduced the finding to a 3-test public-API JUnit; the independent checks followed (see Checks).

Prior reports, history and checks

Prior reports

No prior report found (searched 2026-10-26): 247 logged queries and reads over Lucene's GitHub issues, PRs and commits, Elasticsearch, OpenSearch and Solr, the mailing lists, the JIRA archive and the web, plus the changed files of all 413 open PRs. The nearest items are the introducing PR #1017 and LUCENE-10060, an earlier wrong isCacheable in another class.

Searches6 searches

No issue, PR, review comment, JIRA entry, mailing-list thread or web page says that the shape doc-values box queries stay cacheable after a doc-values update, or that SpatialQuery#queryIsCacheable defaults to true for a doc-values query. Not fixed upstream: the lines are unchanged on upstream main, branch_10x and branch_10_6, and the open PRs that touch these files (#13948, #16847, #709, #16757) do not change cacheability.

Closest, not this bug: #1017, #11690 and #1064 (LUCENE-10654, introduced the code; no caching discussion); #9065 (LUCENE-8017) and #9090 (LUCENE-8042), the origin of the doc-values cacheability rule; LUCENE-10060 (an earlier wrong isCacheable, in DrillSidewaysQuery: a cached instance let deleted documents into its sideways counts; fixed by making its existing isCacheable return false); #13307 and #11973 (other shape doc-values defects); Elasticsearch's own shape doc-values query, which returns DocValues.isCacheable.

Limits: keyword search only; GitHub's search rate limit capped two of the tool runs (GraphQL search covered the same index); private channels cannot be searched.

What was searched, when, and what turned up
Where	When	How	Found
apache/lucene issues and PRs, open and closed	2026-10-26	30 REST queries in three scripted prior-report search runs and 56 GraphQL searches, by identifier (`queryIsCacheable`, `SpatialQuery`, `BaseShapeDocValuesQuery`, `LatLonShapeDocValuesQuery`, `XYShapeDocValuesQuery`, `ShapeDocValuesField`, `newSlowDocValuesBoxQuery`, `DocValues.isCacheable`, LUCENE-10654) and by symptom; a full-text grep of every issue, issue comment and review comment since 2025-10-28	no report; #1017, #11690, #1064 introduced the code
Open PRs	2026-10-26	the changed files of all 413 open apache/lucene PRs, and the patches of the PRs that touch these files	#13948, #16847, #709, #16757 touch the files; none changes cacheability
Commits and history	2026-10-26	6 commit queries; `git log --all -S queryIsCacheable` and the logs of the affected files over the full history	only the introducing commit `d7fd48c9502` and its branch_9x backport `ddf0d0acf4e4`
Elasticsearch, OpenSearch, Solr	2026-10-26	4 tool queries and 12 GraphQL searches; read Elasticsearch's own shape doc-values query	no report; Elasticsearch's query returns `DocValues.isCacheable`
Mailing lists (dev@, java-user@, issues@, commits@)	2026-10-26	68 searches	no report; only commit and review notifications of #1017
JIRA archive and web	2026-10-26	18 JIRA queries plus direct reads of LUCENE-10654, LUCENE-10276 and LUCENE-10060; 15 web queries	no report

History

Introduced with the API in 9.4.0 by LUCENE-10654 (#1017, d7fd48c9502, 2022-08-09); the PR's first commit had the right check and an in-PR refactor dropped it. All 29 releases from 9.4.0 to 10.5.2 and main fail the repro; not fixed upstream.

Versions tested8 rows

Introduced by LUCENE-10654 "Add new ShapeDocValuesField for LatLonShape and XYShape" (#1017, @nknize, issue #11690), merged on main on 2022-08-09 as d7fd48c9502c; the branch_9x backport #1064 (ddf0d0acf4e4) landed the same day, and 9.4.0 (tagged 2022-09-30) is the first release with it. The squash commit added queryIsCacheable (default true), routed SpatialQuery's Weight#isCacheable through it, and added BaseShapeDocValuesQuery without an override; git log --all -S queryIsCacheable over the full history finds only this commit and its branch_9x backport ddf0d0acf4e4. Inside the PR, the first commit had DocValues.isCacheable(ctx, field) and the refactor 2cf24b47c0b6 dropped it (read from the PR's commits, not executed). Bisection is at release level only: 9.3.0 cannot compile the repro because the API is absent, so the bug shipped with the API.

Not executed: 10.6.0 (branch_10_6, released 2026-10-29). Its SpatialQuery lines are identical and BaseShapeDocValuesQuery has no cacheability code, as on branch_10x and upstream main (a95c754ee00, 2026-10-26); branch_10_6 and branch_10x keep the default cache.

Versions and branches executed, with results
Version or commit	Verdict	Result
main `a9779ef3c193`	affected	gradle: 2 of 3 fail with assertions on and off (both shape tests); the control passes
main `a9779ef3c193`, zero configuration	no default cache	the default `IndexSearcher` has no cache since GITHUB#14187: OK (4 = 4); with an installed cache it fails as above
main `ea57360912ed`	affected	2 of 3 fail with assertions on and off, 5 seeds; 40 of 40 shape runs over 20 iterations
10.5.2	affected	JUnit 2 of 3 fail (assertions on and off); zero configuration 5 instead of 4
9.4.0, 9.12.3, 10.0.0	affected	JUnit 2 of 3 fail (the finder's test; assertions off also on 9.4.0 and 10.0.0); zero configuration 5 instead of 4
9.4.1 to 9.12.2, 10.1.0 to 10.5.1 (25 releases)	affected	JUnit 2 of 3 fail with assertions on, the same message in each (the finder's test: `ctx -> true` where the attached test has `_ -> true`)
9.3.0	API absent	the repro does not compile (no shape doc-values field or query); the control passes
9.4.0, 9.12.3, 10.0.0, 10.5.2, main + the fix	not affected	3 of 3 pass; on main also 63 suites, 547 tests

Independent checks

An independent reproducer, an intended-behaviour devil's advocate and an adjudication call it real and re-ran tests on main and released jars; read-only reviews by another model family (OpenAI GPT-5.6) agree on the root cause, one returning a procedural "not real" because it could not run code. This doc's gradle runs reproduced it again.

Checks and verdicts455 words

The independent reproducer, the devil's advocate, the adjudication, the registry re-run and this doc ran tests on stock jars with assertions on and off, on main ea57360912ed and a9779ef3c193 and on release jars. The cross-model reviews were read-only: they could not run any code, so separate runs re-ran the finder's repro for them. The follow-up recommended a refute-lens review that can execute code; it has not been done.

Each independent check and its verdict
Check	Verdict	What it found
finder	candidate	Bytecode scan, differential sweep (35 cases, 8 STALE) and TLA+ model; the 3-test JUnit fails 2 of 3 on main (assertions on and off), 10.5.2, 9.12.3 and 9.4.0; 40 of 40 shape runs over 20 iterations; zero configuration stale on 10.5.2 and 9.12.3; the fix passes, with 31 upstream shape doc-values tests.
review, refute lens	not real (low), procedural	OpenAI GPT-5.6, read-only, could not run Java: agreed on the root cause, found the inputs within the contract and rated the intended argument weak; its "not real" only records that it could not reproduce. A separate run re-ran the repro: 2 of 3 fail.
Independent reproducer	real	Own JUnit outside Lucene's packages: `[0, 1, 2, 3, 4]` instead of `[1, 2, 3, 4]` on main (assertions on and off), 10.5.2, 9.12.3 and 9.4.0; control passes; zero-configuration `SearcherManager` run 5 versus 4 on 10.5.2 and 9.12.3; its own fix passes.
Review, works-as-designed lens	real (medium)	OpenAI GPT-5.6, read-only: the intended argument is weak (its strongest points: experimental APIs, no default cache on main). The verifier re-ran the repro on main, 10.5.2 and 9.4.0 (fail) and with the fix (pass).
Intended-behaviour devil's advocate	real	Rejected a six-part intended argument; found the lost check in PR #1017's commits; re-ran main (assertions on and off), 10.5.2 and the zero-configuration run.
adjudication	confirmed (high)	Re-ran the independent reproduction's JUnit with a new seed (2 of 3 fail, assertions on and off), the finder's with assertions off, and the independent reproduction's fix check (pass).
Follow-up and registry re-run	history, prior art new, keep	29-release matrix, zero-configuration runs, the fix on five targets, the FILTER/MUST_NOT probe and its controls; re-run: the finder's JUnit and the independent reproduction's fail 2 of 3, the fix passes 3 of 3.
security re-evaluation (2026-10-27)	not security-sensitive	Checkers: the stale entry needs the application's own updates, a reopen and a surviving cache; no attacker trigger and no broken documented guarantee for untrusted input.
this doc's runs	reproduced, fix validated	Gradle on `a9779ef3c193`: 2 of 3 fail with assertions on and off, 3 of 3 pass with the fix, 547 tests pass; the 10.5.2 jars: the JUnit fails, the zero-configuration run is stale, both pass with the fix.

Diagram file

The diagram as a still image, generated with AI and checked against the code. It is under 10 MB, the limit for a GitHub attachment.

PNG 586 KB

Sources and provenance

Finding record, Lucene and upstream sources, repro files and diagram notes1,316 words

Finding record

LFV-308, snapshot of 2026-11-06

Registry title: LatLonShape/XYShape newSlowDocValuesBoxQuery reports Weight#isCacheable() == true although it reads the shape from binary doc values, so after an in-place doc-values update the query cache keeps returning the old matches
Area: core / document: shape doc-values queries (BaseShapeDocValuesQuery via SpatialQuery) and the query cache
Priority: Medium, rank 204: The same stale-cache class as LFV-320 and LFV-336, here for in-place-updated experimental doc-values-only shapes and the slow doc-values box query, gets the same tier and ends that group.
Impact class: silent wrong results
Impact note: Applications that store shapes in doc-values-only LatLonShape or XYShape fields (an experimental API), move them in place with IndexWriter#updateDocValues or updateBinaryDocValue, and filter or count with newSlowDocValuesBoxQuery through a query cache that survives the reopen are hit; on 9.4.0-10.6.0 the default IndexSearcher cache is enough (zero-configuration repro executed on a 12,000-doc segment; 10.6.0 by reading), on main only an application-installed cache (no default cache since GITHUB#14187); the moved document keeps its old match decision until the segment is merged into a new one or the entry is evicted; ordinary scale.
Summary: You keep a shape in a doc-values-only field (LatLonShape.createDocValueField or XYShape.createDocValueField), run LatLonShape/XYShape.newSlowDocValuesBoxQuery through a query cache, then move one document's shape out of the box with IndexWriter#updateDocValues and reopen the reader. You expect the reopened searcher to stop matching that document, because a query that reads doc values must report itself not cacheable once those doc values have been updated (SegmentCacheable). Instead its Weight#isCacheable stays true (inherited from SpatialQuery#queryIsCacheable, which always returns true), the segment keeps its core cache key, and the cache keeps serving the pre-update result (5 hits instead of 4 in the repro) until the segment is merged away or the entry is evicted.
Introduced: d7fd48c9502c (LUCENE-10654 'Add new ShapeDocValuesField for LatLonShape and XYShape', apache/lucene#1017 by nknize, main; branch_9x backport ddf0d0acf4e4, #1064) [read: git log -S queryIsCacheable over the full-history clone finds only this commit; it changed SpatialQuery's Weight#isCacheable from 'return true' to the new hook queryIsCacheable (default true) and added BaseShapeDocValuesQuery without an override; exec: release level only, 9.3.0 cannot compile the repro (API absent) and 9.4.0 fails]; inside the PR, the first commit 3554306677 had the shape doc-values query return DocValues.isCacheable(ctx, field) and the refactor commit 2cf24b47c0b6 (2022-07-20) dropped it [read: read-only GitHub API, those commits not executed]; not a regression in any release: the API shipped with the defect (2022-08-09 (#1017 merged on main; #1064 backport the same day); in-PR refactor commit 2022-07-20)
First affected version: 9.4.0 [exec] (first release with the API; 9.3.0 gives compile_error because the API is absent [exec])
Affected versions: all 29 releases from 9.4.0 to 10.5.2 (17 on 9.x, 12 on 10.x) and main ea57360912ed with an installed LRUQueryCache [exec, -ea: both shape tests fail, the binary doc-values control passes; -da also on 9.4.0, 10.0.0, 10.5.2 and main, same result]; with zero configuration (default IndexSearcher cache and caching policy, 12,000-doc segment) 9.4.0, 9.12.3, 10.0.0 and 10.5.2 [exec], not main, which has had no default query cache since GITHUB#14187 (unreleased) [exec]; 10.6.0 (branch_10_6) not executed, same source [read]; not fixed upstream: the lines are unchanged on upstream main e1c4184cb983, branch_10x and branch_10_6 [read, 2026-10-26]
Status: documented (brief, checks and a per-finding doc exist)
Verification: Confirmed by AI-run checks: an independent repro, a cross-model review that could only read the code (its low-confidence “not real” is procedural), an intended-behaviour check, a prior-report search and a skeptical audit; not yet triaged by a person.
Found by: semi-formal; also reached by TLA+/TLC
Upstream and prior reports: none found (2026-10-26, the prior-report record verdict new: 247 logged queries and reads: a scripted prior-report search, 3 runs, with 30 GitHub issue/PR, 6 commit, 4 downstream-repo, 68 mailing-list (dev@, java-user@, issues@, commits@), 18 JIRA-archive and 15 distinct web queries; 56 read-only GraphQL searches over apache/lucene and 12 over Elasticsearch, OpenSearch and Solr; a full-text grep of every apache/lucene issue, issue comment and review comment since 2025-10-28; the changed-file lists of all 413 open PRs; direct reads of the introducing threads and commits; the independent reproduction's and the intended-behaviour check's searches agree); not fixed upstream (SpatialQuery.java:146-148 and :221-222 unchanged on upstream main e1c4184cb983, branch_10x and branch_10_6; the open PRs that touch these files, #13948, #16847, #709 and #16757, do not change cacheability). Related, not reports of this bug: #1017, #11690 and #1064 (LUCENE-10654, introduced the code; the first PR commit 3554306677 had the DocValues.isCacheable check, the in-PR refactor 2cf24b47c0 dropped it, and none of the 10 comments, 30 review comments or reviews mention caching), #9065 (LUCENE-8017) and #9090 (LUCENE-8042) (origin of the doc-values cacheability contract), LUCENE-10060 (the same pattern in DrillSidewaysQuery, fixed by overriding isCacheable), #13307 and #11973 (other shape doc-values defects), Elasticsearch's own ShapeDocValuesQuery (an independent implementation that returns DocValues.isCacheable)
Lucene commit tested: main a9779ef3c193 (commit of 2026-10-26; tested 2026-10-27) through gradle in a scratch worktree, with assertions on and off; the 10.5.2 release jars through a standalone runner (the JUnit with assertions on and off; the zero-configuration main() with assertions off). The earlier checks ran on main ea57360912ed, whose cited lines are identical, on all 29 releases from 9.4.0 to 10.5.2, and on 9.3.0 (API absent). The fix was validated in the scratch worktree: the test with assertions on and off, 63 suites (547 tests) covering the org.apache.lucene.document package and TestLRUQueryCache, TestSegmentCacheables and TestUsageTrackingFilterCachingPolicy with assertions on, and the format, ecjLint and forbiddenApis checks of lucene/core; on the 10.5.2 jars with the fixed class shadowed in.

Lucene and upstream sources

Finding record: lucene/core/src/java/org/apache/lucene/document/SpatialQuery.java:58 (@lucene.internal), :146-148 (queryIsCacheable returns true), :221-222 (Weight#isCacheable returns queryIsCacheable(ctx)); BaseShapeDocValuesQuery.java:38 (@lucene.experimental), :64 (reader.getBinaryDocValues(field)), no queryIsCacheable override (grep of lucene/ finds only SpatialQuery.java:146 and :222)
lucene/core/src/java/org/apache/lucene/search/SegmentCacheable.java:27-35 ('Objects that depend on doc values should return DocValues#isCacheable ... Updated doc values fields are not suitable for caching'); lucene/core/src/java/org/apache/lucene/index/LeafReader.java:63-68 (the core cache helper ignores deletions and doc values updates); lucene/core/src/java/org/apache/lucene/search/LRUQueryCache.java:816-834 (trusts in.isCacheable(context), then keys on getCoreCacheHelper())
lucene/core/src/java/org/apache/lucene/document/LatLonShape.java:89-91 (createDocValueField 'without creating indexable fields'), :263 (newSlowDocValuesBoxQuery); XYShape.java:214 (newSlowDocValuesBoxQuery); ShapeDocValuesField.java:38 (@lucene.experimental); lucene/core/src/java/org/apache/lucene/index/IndexWriter.java:2081-2126 (buildDocValuesUpdate accepts BINARY Field updates; verifyOrCreateDvOnlyField at :2097)
lucene/core/src/java/org/apache/lucene/document/LatLonDocValuesBoxQuery.java:164-165, LatLonDocValuesQuery.java:169-170, XYDocValuesPointInGeometryQuery.java:141-142, BinaryRangeFieldRangeQuery.java:146-147 (sibling doc-values queries return DocValues.isCacheable(ctx, field)); lucene/core/src/test/org/apache/lucene/search/TestLRUQueryCache.java:2222 (testDocValuesUpdatesDontBreakCache)
The finding in 30 seconds: a Lucene main checkout (a9779ef3c193) lucene/core/src/java/org/apache/lucene/search/LRUQueryCache.java:810-834, :891-928
a Lucene main checkout lucene/core/src/java/org/apache/lucene/index/LeafReader.java:65-78
a Lucene main checkout lucene/core/src/java/org/apache/lucene/search/SegmentCacheable.java:24-36
a Lucene main checkout lucene/core/src/java/org/apache/lucene/index/DocValues.java:340-346
a Lucene main checkout lucene/core/src/java/org/apache/lucene/document/LatLonShape.java:89-92, :262-284; XYShape.java:211-217; ShapeDocValuesField.java:28-52
a Lucene main checkout lucene/core/src/java/org/apache/lucene/document/BaseShapeDocValuesQuery.java:34-98; SpatialQuery.java:54-60, :146-148, :209-224
a Lucene main checkout lucene/core/src/java/org/apache/lucene/index/IndexWriter.java:2070-2127
a Lucene main checkout lucene/core/src/java/org/apache/lucene/search/IndexSearcher.java:976-983 (cache only when scores are not needed); BooleanWeight.java:56-59; BooleanClause.java:84-86
Who is affected: a Lucene main checkout lucene/core/src/java/org/apache/lucene/index/FieldInfos.java:562-647 (verifyOrCreateDvOnlyField; points check :619-626); IndexWriter.java:2097
a Lucene clone releases/lucene/10.5.2: lucene/core/src/java/org/apache/lucene/search/IndexSearcher.java:96-99, LRUQueryCache.java:175-198, UsageTrackingQueryCachingPolicy.java:33-54, :122-137
a Lucene main checkout lucene/core/src/java/org/apache/lucene/search/IndexSearcher.java:82-84; lucene/CHANGES.txt:333 (GITHUB#14187)
a Lucene clone origin/branch_10_6 and origin/branch_10x: IndexSearcher.java:99 (default cache) (read 2026-10-27)
a Lucene main checkout lucene/core/src/java/org/apache/lucene/search/IndexSearcher.java:976-983; BooleanWeight.java:56-59; LRUQueryCache.java:252, :277-296, :450-463, :792-808; a Lucene clone releases/lucene/9.4.0 and 9.6.0: LRUQueryCache.java (9.4.0 :150, :170-171)
Why it looks unintended: a Lucene main checkout lucene/core/src/java/org/apache/lucene/search/SegmentCacheable.java:30-32; QueryCache.java:27-34
a Lucene main checkout lucene/core/src/test/org/apache/lucene/search/TestLRUQueryCache.java:2222-2290; TestSegmentCacheables.java:43-92
a Lucene main checkout lucene/core/src/java/org/apache/lucene/document/LatLonDocValuesBoxQuery.java:164-165, LatLonDocValuesQuery.java:169-170, XYDocValuesPointInGeometryQuery.java:141-142, BinaryRangeFieldRangeQuery.java:146-147
Suggested fix: a Lucene main checkout lucene/core/src/java/org/apache/lucene/index/FieldInfos.java:619-626; DocValues.java:340-346; grep 'extends SpatialQuery' over lucene/core/src/java
History: GitHub apache/lucene commits 355430667771d9f3, 2cf24b47c0b6 (read-only API, 2026-10-27)
a Lucene clone origin/branch_10_6 (2763db11b0a), origin/branch_10x (1309534296e), origin/main (a95c754ee00): SpatialQuery.java:146-148, :222; BaseShapeDocValuesQuery.java; IndexSearcher.java (read 2026-10-27)

Other statements trace to the hunt's working files (models, logs, reviews and briefs), which are not published.

Repro files

TestShapeDocValuesQueryCachedAfterUpdate.java (JUnit test): repro/LFV-308/src/org/apache/lucene/document/TestShapeDocValuesQueryCachedAfterUpdate.java (71 lines)

About the diagrams

A step-by-step picture of a separate small example executed on main a9779ef3c193 for the diagram (six point shapes in a doc-values-only field, a box over latitude and longitude 0-10; after doc 0 moves out the cached count is 3 instead of 2, and in a second case a document moved in is missed, 3 instead of 4), not the repro's 20-document layout.

assets/LFV-308/bug.png (586 KB)

Built 2026-11-06 from the hunt's record of this finding and its doc file. Lucene main under test in this hunt: ea57360912ed. Every listed finding was re-run on Lucene main e1faeba08bde on 2026-11-02.