BinaryPoint.newRangeQuery accepts bounds with different numbers of dimensions when their byte totals match

The finding in 30 seconds

The constructor compares the two bounds' packed byte lengths, not their numbers of dimensions, so a shape mistake that keeps the byte total gets through.

Background: how a BinaryPoint range is built

A BinaryPoint field stores points of N dimensions, each dimension a fixed number of bytes; both numbers are fixed when the field is indexed.
newRangeQuery(field, lower, upper) takes each corner of the box as a byte[][], one byte[] per dimension, so lower.length is the number of dimensions.
Inside, the query keeps each corner as one flat byte[] (its dimensions concatenated) plus a dimension count, and cuts the flat bytes back into dimensions of bytes / count bytes.

Diagram of the bug: BinaryPoint.newRangeQuery is called with a lower bound of two 2-byte dimensions, {0,0},{0,0}, and an upper bound of one 4-byte dimension, {1,1,1,1}. Each bound is packed on its own into 4 bytes. The PointRangeQuery constructor receives the two 4-byte arrays and the lower bound's dimension count, 2; it checks only that the byte lengths are equal (4 = 4) and sets 2 bytes per dimension, so the upper bound is read as (1 1),(1 1). The javadoc promises an IllegalArgumentException because 2 != 1 dimensions, but none is thrown. A 2 x 2 against 1 x 3 call, 4 against 3 bytes, is rejected. — Each bound is flattened to bytes on its own: 2 × 2 bytes and 1 × 4 bytes both give 4 bytes. `PointRangeQuery` compares only those lengths (4 = 4) and cuts the upper bound's bytes by the lower bound's layout, so `{1,1,1,1}` becomes `(1 1),(1 1)` and no exception is thrown. Select the still to open it at full size. Generated with AI, checked against the code.

The finding, step by step

On a BinaryPoint field with 2 dimensions of 2 bytes each, build a range but get the upper bound's shape wrong: newRangeQuery("f", {{0,0},{0,0}}, {{1,1,1,1}}). The lower bound has 2 dimensions × 2 bytes, the upper bound 1 dimension × 4 bytes; both are 4 bytes in total.
You expect IllegalArgumentException: the javadoc promises it whenever lowerValue.length != upperValue.length, here 2 != 1.
Instead you get a query, f:[binary(0 0) TO binary(1 1)],[binary(0 0) TO binary(1 1)]. Each bound is flattened to bytes on its own (4 and 4), only those lengths are compared, and the upper bound's 4 bytes are read as two 2-byte dimensions.
On a field indexed as 2 × 2 bytes the query then runs with no error; on a 1 × 4-byte field the search fails later, with a message about the field. With unequal byte totals (2 × 2 against 1 × 3 bytes) the call is rejected, as documented.

Who is affected

Only applications that call the multi-dimensional BinaryPoint.newRangeQuery with a mis-shaped bound whose byte total still matches. The mistake is not reported where the javadoc promises: the query runs silently if the lower bound matches the field's layout; on a field with another layout, search fails later with an error about the field. Well-formed queries and fixed-width point types are unaffected.

Mechanism, terms and numbers608 words

The n-dimensional BinaryPoint.newRangeQuery(String, byte[][], byte[][]) documents @throws IllegalArgumentException ... if lowerValue.length != upperValue.length, that is, when the two bounds have different numbers of dimensions. It packs each bound into one flat byte[] on its own and passes only the lower bound's dimension count to PointRangeQuery, whose constructor compares just the two packed lengths. A dimension-count mismatch is therefore caught only when it also changes the byte total: 2 × 2 bytes against 1 × 4 bytes is accepted, and the upper bound's bytes are read in the lower bound's layout. The caller's shape error then goes unreported: the query runs silently when the lower bound's layout matches the indexed field and, on a segment whose field has a different layout, fails at search time with an IllegalArgumentException about the field. Well-formed queries, mismatches with different byte totals and the fixed-width point types are unaffected. It has been there since the 2016 refactor to packed bounds (LUCENE-7117/7118), in every release from 6.0.0 through 10.6.0 and on main. Low priority; the fix is a single dimension-count check before packing.

The mechanism (main `5adff78bb168`)

The factory. BinaryPoint.newRangeQuery(String, byte[][], byte[][]) (BinaryPoint.java:192-194) returns new PointRangeQuery(field, pack(lowerValue).bytes, pack(upperValue).bytes, lowerValue.length) {...}: each bound is packed separately, and only the lower bound's dimension count is passed on.
pack (BinaryPoint.java:86-119) rejects a null bound and zero dimensions (:87-92) and returns a one-dimension bound as is (:93-95); only for two or more dimensions does it also reject null or empty values and dimensions of unequal byte length within one bound (:96-113), then concatenate them (:114-118). It never sees the other bound.
The constructor (PointRangeQuery.java:72-98) checks numDims > 0 (:75), that lowerPoint.length is a multiple of numDims (:81) and that the two packed lengths are equal (:84-90), then sets bytesPerDim = lowerPoint.length / numDims (:92). With 4 and 4 bytes and numDims = 2, every check passes and bytesPerDim = 2.
Why only BinaryPoint. IntPoint, LongPoint, FloatPoint and DoublePoint carry the same @throws sentence (IntPoint.java:212, LongPoint.java:231, FloatPoint.java:241, DoublePoint.java:241), but their bytes per dimension are fixed, so a different dimension count always changes the byte total and :84 rejects it. The same holds for the sandbox types BigIntegerPoint (16 bytes, BigIntegerPoint.java:51, :217) and HalfFloatPoint (2 bytes, HalfFloatPoint.java:61, :369). The one-dimension overload newRangeQuery(String, byte[], byte[]) wraps each bound in a one-element array (BinaryPoint.java:177-179) and is unaffected.

The shapes (dimensions × bytes per dimension)

lower	upper	packed bytes	result on main
2 × 2	1 × 4	4 / 4	accepted: `f:[binary(0 0) TO binary(1 1)],[binary(0 0) TO binary(1 1)]`
1 × 4	2 × 2	4 / 4	accepted: `f:[binary(0 0 0 0) TO binary(1 1 1 1)]`
2 × 2	4 × 1	4 / 4	accepted, printed as the first row
3 × 2	2 × 3	6 / 6	accepted (an independent test)
2 × 2	1 × 3	4 / 3	rejected: `lowerPoint has length=4 but upperPoint has different length=3`

At search time

One document with a 2 × 2-byte field and a 1 × 4-byte field: the 2 × 2 / 1 × 4 query on the 2 × 2 field counts 1 hit with no error and equals() the well-formed query from {{0,0},{0,0}} to {{1,1},{1,1}}; on the 1 × 4 field the search throws field="p1x4" was indexed with numIndexDimensions=1 but this query has numDims=2. The 1 × 4 / 2 × 2 query behaves the same way the other way round.

Exposure and likelihood229 words

Who. Applications that build multi-dimensional BinaryPoint ranges from bounds they assemble themselves and get one bound's shape wrong while keeping the same byte total. Lucene's query parsers do not build BinaryPoint ranges: outside tests, BinaryPoint appears in main only in BinaryPoint.java, in javadoc (PointValues, SortedDocValuesField, SortedSetDocValuesField) and in the test framework.
Effect. The documented IllegalArgumentException at construction is missing. When the lower bound's layout matches the indexed field, the query runs over the upper bound's bytes in that layout and nothing ever reports the mistake; on a segment whose field has a different layout the search fails with an IllegalArgumentException about the field's dimension count or bytes per dimension (PointRangeQuery.java:627-644), which points away from the bounds; a segment without points for the field gives no hits and no error (:622-625), as for any query.
Default vs opt-in. BinaryPoint is an opt-in field type; no setting is involved, and assertions on or off make no difference.
Scale. One call; nothing is written to the index; no persistent effect and no resource amplification was found.
Likelihood: contrived. One plausible slip, passing one bound as a single already-packed byte[], either re-splits into the intended bound (the query equals() the well-formed one) or fails at search time on a segment that has the field. No public report of it was found since 2016.

Minimal repro

One JUnit test method: build a range from a 2 × 2-byte lower bound and a 1 × 4-byte upper bound and expect the documented IllegalArgumentException. Public API, no index, deterministic; it fails with assertions on and off.

TestBinaryPointRangeDimensionCount.java (one JUnit test, lucene/core tests)30 lines

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package org.apache.lucene.document;

import org.apache.lucene.tests.util.LuceneTestCase;

public class TestBinaryPointRangeDimensionCount extends LuceneTestCase {

  public void testDifferentDimensionCountsRejected() {
    byte[][] lower = {{0, 0}, {0, 0}};
    byte[][] upper = {{1, 1, 1, 1}};
    // expected: the documented IllegalArgumentException; actual: a 2-dim query is built
    expectThrows(
        IllegalArgumentException.class, () -> BinaryPoint.newRangeQuery("f", lower, upper));
  }
}

Command

cp $REPRO/TestBinaryPointRangeDimensionCount.java $LUCENE/lucene/core/src/test/org/apache/lucene/document/
cd $LUCENE && ./gradlew :lucene:core:test --tests org.apache.lucene.document.TestBinaryPointRangeDimensionCount -Ptests.seed=DEADBEEF
# the same failure with -Ptests.asserts=false

Expected: IllegalArgumentException, because the lower bound has 2 dimensions and the upper bound 1.
Observed: No exception: AssertionFailedError: Expected exception IllegalArgumentException but no exception was thrown, with assertions on and off.

All files of this repro: repro/LFV-1226/

More files, environment, full output108 words

TestBinaryPointRangeDimensionCount (one method, extends LuceneTestCase, package org.apache.lucene.document of the lucene/core tests). No index, fixed input, no randomness; the test method itself takes about 0.02 s. Seed DEADBEEF here; runs with five other seeds fail the same way.

As a regression test the same method fits in TestPointQueries next to testInvalidPointLength. The file passes checkGoogleJavaFormat.

Environment

lucene	`main 5adff78bb168 (12.0.0-SNAPSHOT): fails; 6.0.0 to 10.6.0 (nine releases run): fails`
jdk	`OpenJDK 25.0.3 (build 25.0.3+9-LTS)`
os	`Linux aarch64 (nothing platform-specific)`

Observed output

TestBinaryPointRangeDimensionCount > testDifferentDimensionCountsRejected FAILED
    junit.framework.AssertionFailedError: Expected exception IllegalArgumentException but no exception was thrown
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:94A4FB7F3B7B4E15]:0)
        at app//org.apache.lucene.tests.util.LuceneTestCaseParent.expectThrows(LuceneTestCaseParent.java:1432)
        at app//org.apache.lucene.tests.util.LuceneTestCaseParent.expectThrows(LuceneTestCaseParent.java:1418)
        at app//org.apache.lucene.document.TestBinaryPointRangeDimensionCount.testDifferentDimensionCountsRejected(TestBinaryPointRangeDimensionCount.java:27)

# -Ptests.asserts=false: the same failure

Why it looks unintended

The javadoc names this input: @throws IllegalArgumentException ... if lowerValue.length != upperValue.length, on byte[][] bounds, so dimension counts.
The check worked when the clause was written; the 2016 move to packed bounds dropped it without discussion.
No test or CHANGES entry relaxes it; the fixed-width point types still enforce it.

The full argument177 words

Contract. BinaryPoint.java:188-189: @throws IllegalArgumentException if {@code field} is null, if {@code lowerValue} is null, if {@code upperValue} is null, or if {@code lowerValue.length != upperValue.length}. Nothing narrows it.
History. The clause comes from 099e0311398 (LUCENE-7045, 2016-02-23), when PointRangeQuery still took byte[][] bounds and threw lowerPoint has length=... but upperPoint has different length=... on a dimension-count mismatch. e1a1dbfabcc (LUCENE-7117/7118, 2016-03-20) moved to packed bounds and kept the javadoc; its new check first compared upperPoint.length with itself, and 54e662b6dab (later the same day) made it today's packed-length comparison. The review thread (#8173) discusses only that self-comparison.
Tests. No test passes bounds of different dimension counts to BinaryPoint. TestPointQueries.testInvalidPointLength (:2381-2394) asserts the packed-length error on the expert constructor (4 against 8 bytes); testWrongNumDims (:1147) and testWrongNumBytes (:1176) cover a query whose shape disagrees with the indexed field.
Executed bisect. With the parent commit's BinaryPoint.java and PointRangeQuery.java compiled over the 6.0.0 jars the repro passes (lowerPoint has length=2 but upperPoint has different length=1); with the introducing commit's files it fails.

The devil's-advocate check102 words

Strongest argument: this is malformed caller input; the accepted query has a consistent byte-level meaning (it equals() the well-formed query on the same packed bytes), and a layout that does not match the field fails at search time anyway. Why it fails: the javadoc promises to reject exactly this input at construction, the promise was true when it was written and was lost as a side effect of a refactor, and in the silent case nothing ever reports the mistake. The argument lowers the impact (hence Low), not the intent. A committer may still prefer to narrow the javadoc instead (option B).

Suggested fix

Recommended Reject differing dimension counts in newRangeQuery before packing (A, recommended, validated)

Check the dimension counts before packing: in newRangeQuery(String, byte[][], byte[][]), throw IllegalArgumentException when lowerValue.length != upperValue.length. The repro passes, and so do the 54 TestPointQueries tests.

  public static Query newRangeQuery(String field, byte[][] lowerValue, byte[][] upperValue) {
    if (lowerValue != null && upperValue != null && lowerValue.length != upperValue.length) {
      throw new IllegalArgumentException(
          "lowerValue has "
              + lowerValue.length
              + " dimensions but upperValue has "
              + upperValue.length);
    }
    return new PointRangeQuery(
        field, pack(lowerValue).bytes, pack(upperValue).bytes, lowerValue.length) {

All options and trade-offs2 options · 439 words

Option A was built and checked through gradle on 5adff78bb168; B is documentation only and was not run. A restores the documented behaviour at no hot-path cost; B makes the javadoc match the code but leaves the caller's shape error silent.

Reject differing dimension counts in newRangeQuery before packing (A, recommended, validated)

In BinaryPoint.newRangeQuery(String, byte[][], byte[][]) (BinaryPoint.java:192), throw IllegalArgumentException when both arrays are non-null and lowerValue.length != upperValue.length. Null arguments keep pack's existing IllegalArgumentException. With equal dimension counts and equal packed lengths, pack's per-bound check already forces equal bytes per dimension, so nothing else is needed.

  public static Query newRangeQuery(String field, byte[][] lowerValue, byte[][] upperValue) {
    if (lowerValue != null && upperValue != null && lowerValue.length != upperValue.length) {
      throw new IllegalArgumentException(
          "lowerValue has "
              + lowerValue.length
              + " dimensions but upperValue has "
              + upperValue.length);
    }
    return new PointRangeQuery(
        field, pack(lowerValue).bytes, pack(upperValue).bytes, lowerValue.length) {

Trade-offs

One comparison per query construction. Code that relies on the silent re-split today (for example an upper bound passed as one packed byte[] against a multi-dimension lower bound) gets the exception the javadoc already documents. One message changes: for a null field together with two non-null bounds of different dimension counts, the new dimension-count message now comes before PointRangeQuery.checkArgs' field must not be null (PointRangeQuery.java:73, :107-108); both are IllegalArgumentException. Starting the fix with PointRangeQuery.checkArgs(field, lowerValue, upperValue), as IntPoint.newRangeQuery does (IntPoint.java:216), would keep the null-field message first (not run).

What the checks validated

Gradle in a scratch worktree at 5adff78bb168, seed DEADBEEF: the repro fails without the fix (assertions on and off) and passes with it (assertions off).
With the fix and assertions on: the repro (1) and TestPointQueries (56, of which the 2 @Nightly tests are skipped): 57 tests, 0 failures. checkGoogleJavaFormat passes.
Earlier the same change was checked by compiling the patched class over main: the repro passed with assertions on and off, the independent 3 × 2 against 2 × 3 test passed, and the 54 TestPointQueries tests passed.

Narrow the @throws clause to the packed byte lengths (B, documentation only)

Change the @throws text of newRangeQuery(String, byte[][], byte[][]) to say what is enforced: the two bounds' total byte lengths must be equal. The expert PointRangeQuery constructor's javadoc (PointRangeQuery.java:69-70) also still says lowerValue.length != upperValue.length, names that constructor does not have; that stale clause can be fixed with either option.

Trade-offs

No behaviour change, so nothing can break, but the caller's shape mistake stays silent whenever the byte totals agree; the fixed-width point types do not need the change. Not run.

Downstream note

Lucene's query parsers never build BinaryPoint ranges, so only a downstream product that turns its own input into multi-dimensional BinaryPoint bounds can reach this, and checking that both bounds have the same number of dimensions there avoids it.

How it was found

A proof-obligation audit of BinaryPoint.newRangeQuery's documented @throws contract: every input the javadoc says it rejects must be rejected. The obligation fails for bounds with different dimension counts but equal byte totals, and a one-method JUnit test witnessed it on main and on the release jars.

The method in full126 words

Method: semi-formal (a proof-obligation audit of a class's contract, witnessed by an executable repro).

The audit takes each @throws condition of newRangeQuery(String, byte[][], byte[][]) as an obligation and traces which check discharges it: the null conditions are discharged by pack (BinaryPoint.java:87-89) and PointRangeQuery.checkArgs (PointRangeQuery.java:106-116); lowerValue.length != upperValue.length by nothing, because only the packed lengths reach PointRangeQuery.java:84.
A shape probe then mapped which mismatches get through (equal byte totals) and which do not (unequal totals), and a one-method JUnit test plus the one-check fix that makes it pass witnessed the gap.
An independent reproducer, an executed cross-model refute review and a devil's-advocate verifier reproduced it with their own tests or re-runs, and an adjudication re-ran the test ("Prior reports, history and checks").

Prior reports, history and checks

Prior reports

No prior report found (searched 2026-11-03: GitHub issues, PRs and commits incl. the JIRA archive, downstream trackers, mailing lists, the web and the files of all open PRs; re-checked the same day). Nearest: #8173 (the 2016 refactor that created the gap) and #8101 (which added the clause).

Searches3 searches

No report that BinaryPoint.newRangeQuery accepts bounds of different dimension counts with equal byte totals, against its documented IllegalArgumentException, was found, and nothing upstream fixes it. Limits: keyword search only; GitHub search does not index PR diffs or every review comment; private reports are not visible.

What was searched, when, and what turned up
Where	When	How	Found
GitHub issues, PRs and commits (apache/lucene), JIRA archive, mailing lists, downstream trackers, web	2026-11-03	a scripted prior-report search (51 queries over GitHub, commits, Elasticsearch, OpenSearch and Solr, dev@, java-user@, issues@, commits@, JIRA and the web; 170 hits, every high and medium hit and the top 40 read) and 12 manual GitHub searches: BinaryPoint, "lowerValue.length != upperValue.length", "upperPoint has different length", "different number of dimensions", PointRangeQuery dimensions mismatch, BinaryPoint in:title, ...	no report of this defect
Open PRs and recent activity	2026-11-03	the changed files of all 410 open PRs; the 47 issues and PRs updated since 2026-11-01; items created since 2026-11-02 (#16962-#16967); a re-check	none touches BinaryPoint.java; of eight PRs touching PointRangeQuery.java only #13711 touches the constructor checks, and it moves them unchanged
Related items (read)	2026-11-03	#8173 / LUCENE-7118, #8172 / LUCENE-7117, #8101 / LUCENE-7045, #9754 / LUCENE-8708	background only: the refactor that created the gap (its review fixed only the self-comparison), the change that added the clause, and a patch in a comment that quotes the code

History

Introduced by e1a1dbfabcc (LUCENE-7117/7118, #8173, 2016-03-20), before 6.0.0, the first release with BinaryPoint. Every release from 6.0.0 to 10.6.0 is affected (ten targets run, the rest by reading every tag and, for the untagged 8.8.2 to 8.11.3, javap of the release jars); with the parent commit's code the repro passes. Not fixed upstream.

Versions tested9 rows

Before: 099e0311398 (LUCENE-7045, Robert Muir, 2016-02-23) wrote the @throws clause while the byte[][] PointRangeQuery constructor rejected differing dimension counts.

Introduced: e1a1dbfabcc "LUCENE-7117, LUCENE-7118: Remove multidimensional arrays from PointRangeQuery" (Robert Muir, 2016-03-20; a JIRA patch, no PR; migrated as #8173 and #8172), copied the same day to branch_6_0 (d8c4e6977b0, in 6.0.0 and 6.0.1) and branch_6x (51b109620be, 6.1.0 to 6.6.6). It moved to packed bounds with numDims = lowerValue.length; its length check compared upperPoint.length with itself. Ryan Ernst's follow-up 54e662b6dab, later the same day, made it lowerPoint.length != upperPoint.length, the check of every release, which cannot see the dimension count.

Not fixed upstream: upstream main 1c2072f0284 (2026-11-02) has the same factory body and check; the newest commits on the two files are 644feeb02a9 (#13795, 2024-09-17) and 1169cd2007d (#16089, 2026-10-14), neither touching the check.

Before 9.1.0 the test imports LuceneTestCase from org.apache.lucene.util, and before 9.0.0 it runs under the JUnit 4 runner; the call is unchanged.

Versions and branches executed, with results
Version or commit	Verdict	Result
5.5.5	no API	no `BinaryPoint`: the test does not compile
parent of `e1a1dbfabcc`	over 6.0.0 jars	passes: `lowerPoint has length=2 but upperPoint has different length=1`
`e1a1dbfabcc`	over 6.0.0 jars	fails: no exception (even 2 × 2 against 1 × 3 is accepted before the follow-up)
6.0.0	affected	fails, assertions on and off
6.6.6, 7.7.3, 8.11.4, 9.0.0	affected	fails (assertions off)
9.12.3, 10.0.0, 10.5.2	affected	fails
10.6.0	affected	fails, assertions on and off
main 5adff78bb168	JUnit, gradle	fails, assertions on and off
main with fix A	not affected	passes; 54 `TestPointQueries` tests pass

Independent checks

The finder, an independent reproducer with its own 3 × 2 against 2 × 3 test, an executed cross-model refute review (OpenAI GPT-5.6: real, high), a devil's-advocate verifier and an adjudication (confirmed, Low, high confidence) all rated it real, and a final re-run reproduced it. This doc's gradle runs reproduced it on main and validated the fix.

Checks and verdicts326 words

The cross-model review ran on the older main ea57360912ed, where BinaryPoint.java and PointRangeQuery.java are identical to 5adff78bb168. Every other check ran on 5adff78bb168 or release jars.

Each independent check and its verdict
Check	Verdict	What it found
finder	reproduced	Own test fails on main (assertions on and off), 10.6.0 and 9.12.3; the shape probe shows which mismatches get through; with the one-check fix the test and the 54 `TestPointQueries` tests pass.
independent reproducer	real, Low	Own test, 3 × 2 against 2 × 3 bytes (both bounds through `pack`'s multi-dimension branch), fails on main (assertions on and off), 10.6.0, 10.5.2 and 9.12.3; a search-time probe counts 1 hit with no error.
cross-model review, refute lens (OpenAI GPT-5.6, executed)	real (high)	Own tests fail while its controls (matching shapes accepted, unequal byte totals rejected) pass; the accepted query runs and matches one document; with the fix its tests and `TestPointQueries` pass. The "malformed input with deterministic semantics" defence was rated weak against the explicit `@throws`.
devil's-advocate verifier	real (moderately high), Low	Found no by-design statement: the clause was true when written and lost in the 2016 refactor; its probe shows the accepted query `equals()` the well-formed one and the field-mismatch failure, so the case against is about impact, not intent.
adjudication	confirmed, Low, high	Re-ran the test with a new seed, assertions off: fails. Wording: say the upper bound is read in the lower bound's layout and the shape error is not reported, not that the query matches a different box.
final re-run and repro check	reproduced; confirmed, Low	Re-ran the repro on main with assertions off (two more seeds): fails; public API, small ordinary input, deterministic, one test.
this doc's gradle check	reproduced; fix A validated	Repro fails on main with assertions on and off; with fix A it passes, `TestPointQueries` passes (57 tests with the repro, 0 failures, 2 nightly skipped) and `checkGoogleJavaFormat` passes.

Diagram file

The diagram as a still image, generated with AI and checked against the code. It is under 10 MB, the limit for a GitHub attachment.

PNG 448 KB

Sources and provenance

Finding record, Lucene and upstream sources, repro files and diagram notes1,449 words

Finding record

LFV-1226, snapshot of 2026-11-06

Registry title: BinaryPoint.newRangeQuery(String, byte[][], byte[][]) does not throw the IllegalArgumentException its javadoc promises when lowerValue.length != upperValue.length if both bounds pack to the same number of bytes (for example 2 dimensions x 2 bytes against 1 x 4), and instead builds a query that reads the upper bound's bytes in the lower bound's dimension layout, because only the packed byte lengths are compared (since 6.0.0, LUCENE-7117/LUCENE-7118)
Area: core / document: BinaryPoint range query factory (n-dimensional BinaryPoint.newRangeQuery, PointRangeQuery constructor checks)
Priority: Low, rank 889: BinaryPoint.newRangeQuery skips its documented IllegalArgumentException when the two bounds have different dimension counts but the same byte total, in every release since 6.0.0, so the caller's shape error goes unreported and one bound is read in the other's layout. It needs a caller bug on the rarely used variable-width point type with matching byte totals, nothing is persisted, and the trigger is contrived.
Impact class: minor
Impact note: Applications that build a multi-dimensional BinaryPoint range from a lower and an upper bound whose dimension counts differ but whose byte totals agree get no IllegalArgumentException, so the caller's shape error goes unreported: the query reads the upper bound's bytes in the lower bound's layout and runs when that layout matches the indexed field, and otherwise fails only at search time with a different, field-mismatch IllegalArgumentException; query construction only, so any index configuration, -ea and -da, every release from 6.0.0 to 10.6.0 and main; well-formed bounds, bounds whose byte totals differ and the fixed-width point types (IntPoint, LongPoint, FloatPoint, DoublePoint) are unaffected, and nothing is persisted.
Summary: You call BinaryPoint.newRangeQuery("f", lower, upper) with a lower bound of two 2-byte dimensions and an upper bound of one 4-byte dimension, and expect the IllegalArgumentException its javadoc promises 'if lowerValue.length != upperValue.length'. Instead you get a query, printed as f:[binary(0 0) TO binary(1 1)],[binary(0 0) TO binary(1 1)]: the 4-byte upper bound was silently re-split into two 2-byte dimensions. newRangeQuery packs each bound on its own and passes only lowerValue.length as the dimension count, and the PointRangeQuery constructor compares only the packed byte lengths, so a dimension-count mismatch gets through whenever the byte totals agree.
Introduced: e1a1dbfabcc9defb22ba091be1633a31a2810ab8 'LUCENE-7117, LUCENE-7118: Remove multidimensional arrays from PointRangeQuery' by Robert Muir (issues LUCENE-7118 = #8173 and LUCENE-7117 = #8172; JIRA patch, no PR); branch_6_0 copy d8c4e6977b0 (in 6.0.0 and 6.0.1) and branch_6x copy 51b109620be (6.1.0-6.6.6), same day, byte-identical files [read: git blame -w on BinaryPoint.java:193-194 and PointRangeQuery.java:84, git log -S on the @throws clause and on 'upperPoint has different length', git tag --contains; exec: mixed builds over the 6.0.0 jars, where the parent's BinaryPoint.java and PointRangeQuery.java pass (IllegalArgumentException 'lowerPoint has length=2 but upperPoint has different length=1') and the commit's own two files fail]. It replaced the byte[][] PointRangeQuery constructor, which rejected differing dimension counts, with PointRangeQuery(String, byte[], byte[], int numDims) and made BinaryPoint pass pack(lowerValue).bytes, pack(upperValue).bytes, lowerValue.length. Its lower/upper comparison was first the no-op 'upperPoint.length != upperPoint.length'; 54e662b6dab (Ryan Ernst, later the same day, 'LUCENE-7118: Fix packed points upper/lower bound length check') made it the packed-length check 'lowerPoint.length != upperPoint.length' of every release, which cannot see the dimension count. The @throws clause dates from 099e0311398 (LUCENE-7045, Robert Muir, 2016-02-23), written while the byte[][] constructor still enforced it, and has only been re-wrapped since (2016-03-20 (author and commit date 2016-03-20T16:05:17-04:00; branch_6_0 and branch_6x copies the same day; follow-up 54e662b6dab 2016-03-20T13:34:22-07:00); first release 6.0.0)
First affected version: 6.0.0 [exec] (6.0.0 is the first release with BinaryPoint at all; 5.5.5 has no BinaryPoint, so the repro does not compile there [exec]; before e1a1dbfabcc the factory existed only on unreleased master and branches, where the check held, so no release has the n-dimensional factory without the defect)
Affected versions: Every release from 6.0.0 through 10.6.0 (4ae946e5a9b), 93 releases, and main. Executed and reproduced: 6.0.0 (-ea, -da), 6.6.6, 7.7.3, 8.11.4, 9.0.0, 9.12.3, 10.0.0, 10.6.0 (-ea, -da) and main 5adff78bb168 (-ea, -da) with the finder's one-test repro (before 9.1.0 the same test with LuceneTestCase from its older package org.apache.lucene.util and the JUnit 4 runner), and 10.5.2 with the independent reproduction's own test [exec]; all 84 tagged releases from releases/lucene-solr/6.0.0 to releases/lucene/10.6.0 have the same newRangeQuery body and the packed-length-only constructor check [read], and javap of the untagged 8.8.2-8.11.3 jars matches the executed 8.11.4 jar [read]. Not affected: 5.5.5 and older (API absent; 5.5.5 executed: compile error). Not fixed upstream: upstream main 1c2072f0284 (author date 2026-11-02) has the same newRangeQuery body (BinaryPoint.java:192-194) and the same check (PointRangeQuery.java:84) [read, contents API, 2026-11-03]. Not new in 10.6.
Status: documented (brief, checks and a per-finding doc exist)
Verification: Confirmed by AI-run checks: an independent repro, a cross-model review that agreed, an intended-behaviour check, a prior-report search and a skeptical audit; not yet triaged by a person.
Found by: semi-formal: proof-obligation audit
Upstream and prior reports: none found (searched 2026-11-03 and rechecked the same day: a scripted prior-report search with a new spec over apache/lucene issues, PRs and commits, downstream repositories, the dev@, java-user@, issues@ and commits@ lists, the JIRA archive and the web (51 queries, 170 hits: 1 high and 8 medium, all read, and the top 40), the finder's earlier run (51 queries, 805 hits) re-read, 12 manual GitHub issue/PR searches, the changed files of all 410 open apache/lucene PRs (none touches BinaryPoint.java; of the eight that touch PointRangeQuery.java only #13711 touches the constructor checks, and it moves them unchanged), the 47 issues and PRs updated since 2026-11-01, commit searches and git log -S on the @throws clause and the error message, Elasticsearch, OpenSearch, Solr and lucene-solr searches, JIRA queries and 6 web queries; the intended-behaviour check's independent search (42 queries, 155 hits) agrees). Nearest items, not this bug: #8173 (LUCENE-7118, the refactor that created the gap; its only review remark fixed the no-op self-comparison), #8101 (LUCENE-7045, which added the @throws clause), #9754 (term overlap only, a patch quoting the code). Re-checked 2026-11-03: 'BinaryPoint newRangeQuery', 'lowerValue.length != upperValue.length' and 'upperPoint has different length' match only #9754; no issue or PR mentioning BinaryPoint or PointRangeQuery updated since 2026-11-01; items created since 2026-11-02 are #16962-#16967, none about points; newest upstream commits on the fix site by author date are BinaryPoint.java 644feeb02a9 (2024-09-17, #13795) and PointRangeQuery.java 1169cd2007d (2026-10-14, #16089), neither touching the check. Not fixed upstream (main 1c2072f0284).
Lucene commit tested: main 5adff78bb168 (12.0.0-SNAPSHOT) through gradle on 2026-11-03 in a scratch worktree, seed DEADBEEF: the repro without the fix (assertions on and off: fails) and with the fix (assertions off: passes; assertions on, with TestPointQueries: 57 tests, 0 failures, the 2 @Nightly tests skipped), plus checkGoogleJavaFormat. The repro or the same call in an equivalent test also ran on 10.6.0, 10.5.2, 10.0.0, 9.12.3, 9.0.0, 8.11.4, 7.7.3, 6.6.6 and 6.0.0 (all fail) and on 5.5.5 (no BinaryPoint: does not compile). Only Linux aarch64 with OpenJDK 25.0.3 was run.

Lucene and upstream sources

Finding record: lucene/core/src/java/org/apache/lucene/document/BinaryPoint.java:182-191 (at 5adff78bb168) (javadoc of newRangeQuery(String, byte[][], byte[][]): '@throws IllegalArgumentException if {@code field} is null, if {@code lowerValue} is null, if {@code upperValue} is null, or if {@code lowerValue.length != upperValue.length}' at :188-189), :192-194 (body: new PointRangeQuery(field, pack(lowerValue).bytes, pack(upperValue).bytes, lowerValue.length)), :86-119 (pack: validates each bound on its own, never against the other bound)
lucene/core/src/java/org/apache/lucene/search/PointRangeQuery.java:72-98 (at 5adff78bb168) (constructor: numDims > 0, lowerPoint.length % numDims at :81, packed-length check lowerPoint.length != upperPoint.length at :84, bytesPerDim = lowerPoint.length / numDims at :92)
Two Lucene main checkouts: git diff ea57360912ed 5adff78bb168 on BinaryPoint.java and PointRangeQuery.java is empty (2026-11-03)
https://github.com/apache/lucene/issues/8173 (LUCENE-7118 thread), https://github.com/apache/lucene/issues/8101 (LUCENE-7045)
The finding in 30 seconds: lucene main@5adff78bb168 lucene/core/src/java/org/apache/lucene/document/BinaryPoint.java:86-119 (pack), :177-179 (one-dimension overload), :182-194 (javadoc :188-189 and body :192-194)
lucene main@5adff78bb168 lucene/core/src/java/org/apache/lucene/search/PointRangeQuery.java:72-98 (constructor checks :75, :81, :84-90; bytesPerDim :92)
lucene main@5adff78bb168 lucene/core/src/java/org/apache/lucene/document/IntPoint.java:212, LongPoint.java:231, FloatPoint.java:241, DoublePoint.java:241 (the same @throws sentence)
lucene main@5adff78bb168 lucene/sandbox/src/java/org/apache/lucene/sandbox/document/BigIntegerPoint.java:51 (BYTES = 16), :217; HalfFloatPoint.java:61 (BYTES = 2), :369 (the same @throws sentence)
Who is affected: lucene main@5adff78bb168 grep -rlw BinaryPoint lucene (non-test files: BinaryPoint.java, PointValues.java, SortedDocValuesField.java, SortedSetDocValuesField.java, test-framework BasePointsFormatTestCase.java, TestUtil.java)
lucene main@5adff78bb168 lucene/core/src/java/org/apache/lucene/search/PointRangeQuery.java:621-645 (checkValidPointValues: no point values -> false; numIndexDimensions and bytesPerDim checks), :254-258, :348-351 (false -> no scorer, count 0)
Why it looks unintended: lucene main@5adff78bb168 lucene/core/src/java/org/apache/lucene/document/BinaryPoint.java:188-189
lucene main@5adff78bb168 lucene/core/src/test/org/apache/lucene/search/TestPointQueries.java:1147, :1176, :2381-2394
https://github.com/apache/lucene/issues/8173 (LUCENE-7118 thread)
Suggested fix: lucene main@5adff78bb168 lucene/core/src/test/org/apache/lucene/search/TestPointQueries.java:380, :2433 (@Nightly)
lucene main@5adff78bb168 lucene/core/src/java/org/apache/lucene/search/PointRangeQuery.java:69-70 (expert constructor javadoc), :73, :106-109 (checkArgs); lucene/core/src/java/org/apache/lucene/document/IntPoint.java:216
Diagrams: lucene main@5adff78bb168 lucene/core/src/java/org/apache/lucene/document/BinaryPoint.java:86-119, :188-194; lucene/core/src/java/org/apache/lucene/search/PointRangeQuery.java:84-92

Other statements trace to the hunt's working files (models, logs, reviews and briefs), which are not published.

Repro files

TestBinaryPointRangeDimensionCount.java (one JUnit test, lucene/core tests): repro/LFV-1226/TestBinaryPointRangeDimensionCount.java (30 lines)

About the diagrams

The diagram follows the repro and the code at 5adff78bb168 (BinaryPoint.java:86-119, :188-194; PointRangeQuery.java:84-92).

assets/LFV-1226/bug.png (448 KB)

Built 2026-11-06 from the hunt's record of this finding and its doc file. Lucene main under test in this hunt: ea57360912ed. Every listed finding was re-run on Lucene main e1faeba08bde on 2026-11-02.