Lucene's HNSW searcher accepts a graph that reports an unknown maxConn, then uses that value, -1, as an array size.
HnswGraphis the public abstract class that Lucene's HNSW search walks. Lucene's vectors formats implement it, and so can your own code, for example an adapter over another library's graph.maxConn()reports M, the maximum number of links per node. A graph that does not know M can return the public constantHnswGraph.UNKNOWN_MAX_CONN, which is -1.HnswGraphSearcher.searchhas two searchers. The filtered one needs M, so a graph that reports -1 always gets the plain one: "We can only use filtered search if we know the maxConn".
![A light card titled "What the scratch array in an HNSW search is for": a box for an HnswGraph (nodes and their neighbour lists, maxConn() returns M) points to an HnswGraphSearcher box that lists one step of the search (take the best candidate node, put its unvisited neighbours in the scratch array bulkNodes, score them all in one bulkScore call, keep the good ones as results and candidates), whose row of empty slots labelled "its size: maxConn() * 2 slots" is outlined in red and marked as where the bug sits, and that box points to a results box (the k nearest nodes); below, an orange Expected panel shows that with an unknown M the scratch array should grow to fit the neighbours and the search should still return the nearest nodes, next to a blue Actual panel where maxConn() is -1, so the searcher asks for new int[-2] (red, with a cross) and every search throws NegativeArraySizeException before any node is scored.](../assets/LFV-1238/concept.png)
maxConn() = -1 is sent by the search dispatch to the plain searcher, which sizes its scratch as -1 x 2 = -2; new int[-2] throws before any node is scored. Select the still to open it at full size. Generated with AI, checked against the code.- You implement
HnswGraphfor a tiny graph, one node (node 0) with no links, and returnHnswGraph.UNKNOWN_MAX_CONN(-1) frommaxConn(). - You search it with
HnswGraphSearcher.search(scorer, collector, graph, null)for k = 1. - You expect 1 hit, node 0. Lucene 10.2.2 returns it, and so does main when
maxConn()returns 16. - Instead every search throws
NegativeArraySizeException: -2: since 10.3.0 the plain searcher allocates its bulk-scoring scratch asnew int[maxConn() * 2], and -1 x 2 = -2.
Only code that implements its own HnswGraph without knowing M (a custom vectors format, or an adapter over another library's graph), reports UNKNOWN_MAX_CONN, and searches it with HnswGraphSearcher.search, on 10.3.0 to 10.6.0 and main. Lucene's own indexing, merges and kNN queries are not affected. Every such search of a graph with an entry node throws; no wrong hit is returned.
Mechanism, terms and numbers424 words
HnswGraph declares the public constant UNKNOWN_MAX_CONN = -1 for a graph that does not know M, and HnswGraphSearcher.search checks for it explicitly: such a graph skips only the filtered searcher ("We can only use filtered search if we know the maxConn") and gets the plain one. Since 10.3.0 the plain searcher sizes its bulk-scoring scratch as graph.maxConn() * 2, so the first search of a graph that reports the constant and has an entry node runs new int[-2] and throws NegativeArraySizeException: -2, with assertions on and off. Lucene 10.2.x searched the same graph fine. No Lucene format, merge or kNN query reaches the path; only HnswGraph implementations from outside Lucene do, and they fail loudly, never with a wrong hit.
- The constant is public API.
HnswGraphis a public abstract class with a protected constructor in an exported package (HnswGraph.java:53-57,module-info.java:50).maxConn()"returns M, the maximum number of connections for a node" (:88-89), with no lower bound;HnswGraph.EMPTYreturns the constant (:137-140). - The dispatch keeps such graphs on the plain searcher. The filtered searcher is chosen only if
graph.maxConn() != HnswGraph.UNKNOWN_MAX_CONN(HnswGraphSearcher.java:133-139); otherwise a plainHnswGraphSearcheris built (:141-144), with or withoutacceptOrds. - The plain searcher sizes its scratch from maxConn.
searchLevelfirst callsprepareScratchState(size, graph.maxConn() * 2)(:284); for a graph with more than one level,findBestEntryPointmakes the same call before scoring the entry (:223). - The throw.
prepareScratchStateallocatesbulkNodes = new int[bulkScoreSize]when the scratch is missing or too small (:370-373).bulkNodesstartsnull(:42), so the first search runsnew int[-2](:371). The JVM refuses the allocation before any array exists. Only a graph without an entry node (entryNode()= -1) returns earlier, before the sizing (:218-220,AbstractHnswGraphSearcher.java:77-81). - Why Lucene itself never hits it. Every current format reports its stored M (
Lucene99HnswVectorsReader.java:652-653,OnHeapHnswGraph.java:205-206); the reader never runs HNSW search on an empty graph (Lucene99HnswVectorsReader.java:368,:376-378), soHnswGraph.EMPTYis never searched; the Lucene 9.0 backward codec's graph, the only other non-test source of the constant, is searched by its ownLucene90OnHeapHnswGraph.search. One test graph also returns it (TestHnswUtil.MockGraph,TestHnswUtil.java:277-279); it is used only by theHnswUtiltests and never searched.
The same failure happens with acceptOrds set (the filtered searcher is skipped, the plain one throws) and, for a two-level graph, already in the entry-point phase at :223.
Exposure and likelihood213 words
- Who. Code that implements
HnswGraphitself without knowing M and searches it with Lucene'sHnswGraphSearcher. No default path: Lucene's formats always report their stored M. - Effect. Every search of such a graph that has an entry node throws, from the first call, with assertions on and off. Nothing is written out of bounds, the graph and index are unchanged, and no wrong hit is returned.
- Scale. Any graph size; one node is enough. With
acceptOrdsset, and for graphs with more than one level, it throws the same way. - Workaround. Report a real value. The scratch holds
2 * maxConn()entries, so any value of at least half the longest neighbour list, rounded up, avoids the exception; for an HNSW graph its M does (level 0 holds up to2 * Mlinks,HnswGraphBuilder.java:131-132). A real value also makes filtered searches eligible forFilteredHnswGraphSearcherwhen the strategy picks it (HnswGraphSearcher.java:133-139), and that searcher's exploration is computed frommaxConn()(FilteredHnswGraphSearcher.java:51-54): filtered behaviour changes too, and a value that is only a bound, not the graph's M, tunes it from that bound. - Likelihood: rare. One known downstream graph adapter returned the constant for a few days under 10.2.1 and switched to a real value before 10.3.0 shipped; no failure was reported.
One JUnit test method on the public API: a one-node graph whose maxConn() returns UNKNOWN_MAX_CONN, a scorer that scores every node 1.0, and HnswGraphSearcher.search with k = 1. No index, deterministic, single-threaded; it fails with assertions on and off.
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.lucene.util.hnsw;
import java.io.IOException;
import org.apache.lucene.search.DocIdSetIterator;
import org.apache.lucene.search.TopDocs;
import org.apache.lucene.search.TopKnnCollector;
import org.apache.lucene.tests.util.LuceneTestCase;
public class TestHnswSearchUnknownMaxConn extends LuceneTestCase {
// Expected: the single node 0 is returned; actual: NegativeArraySizeException: -2.
public void testSearchGraphWithUnknownMaxConn() throws IOException {
HnswGraph graph =
new HnswGraph() {
@Override
public void seek(int level, int target) {}
@Override
public int size() {
return 1;
}
@Override
public int nextNeighbor() {
return DocIdSetIterator.NO_MORE_DOCS;
}
@Override
public int numLevels() {
return 1;
}
@Override
public int maxConn() {
return UNKNOWN_MAX_CONN;
}
@Override
public int entryNode() {
return 0;
}
@Override
public NodesIterator getNodesOnLevel(int level) {
throw new UnsupportedOperationException();
}
@Override
public int neighborCount() {
return 0;
}
};
RandomVectorScorer scorer =
new RandomVectorScorer() {
@Override
public int maxOrd() {
return 1;
}
@Override
public float score(int node) {
return 1f;
}
};
TopKnnCollector collector = new TopKnnCollector(1, Integer.MAX_VALUE);
HnswGraphSearcher.search(scorer, collector, graph, null);
TopDocs topDocs = collector.topDocs();
assertEquals(1, topDocs.scoreDocs.length);
assertEquals(0, topDocs.scoreDocs[0].doc);
}
}
cp $REPRO/TestHnswSearchUnknownMaxConn.java $LUCENE/lucene/core/src/test/org/apache/lucene/util/hnsw/
cd $LUCENE && ./gradlew :lucene:core:test --tests org.apache.lucene.util.hnsw.TestHnswSearchUnknownMaxConn -Ptests.seed=DEADBEEF
# and with -Ptests.asserts=false (assertions off): the same failure
- Expected
- One hit, node 0, as on 10.2.2 and as on main when
maxConn()returns 16. - Observed
java.lang.NegativeArraySizeException: -2atHnswGraphSearcher.prepareScratchState(HnswGraphSearcher.java:371), with assertions on and off.
More files, environment, full output169 words
Put the file in lucene/core/src/test/org/apache/lucene/util/hnsw/ (it uses no package-private member). It compiles unchanged against 10.2.0 and later, where maxConn() and the constant exist. Replacing UNKNOWN_MAX_CONN with 16 makes it pass on main, so the reported maxConn is the only variable. It is also the regression test for the fix. It covers the sizing; its node has no neighbours, so it does not reach fix A's grow path, which the probe in the fix section exercised.
| lucene | main 5adff78bb168 (12.0.0-SNAPSHOT): fails; 10.3.0, 10.6.0: fail; 10.2.2: passes |
|---|---|
| jdk | OpenJDK 25.0.3 |
| os | Linux aarch64 (nothing platform-specific) |
cp $REPRO/TestHnswSearchUnknownMaxConn.java $LUCENE/lucene/core/src/test/org/apache/lucene/util/hnsw/
cd $LUCENE && ./gradlew :lucene:core:test --tests org.apache.lucene.util.hnsw.TestHnswSearchUnknownMaxConn -Ptests.seed=DEADBEEF
# and with -Ptests.asserts=false (assertions off): the same failure
TestHnswSearchUnknownMaxConn > testSearchGraphWithUnknownMaxConn FAILED
java.lang.NegativeArraySizeException: -2
at __randomizedtesting.SeedInfo.seed([DEADBEEF:2B014FD292B58A08]:0)
at org.apache.lucene.util.hnsw.HnswGraphSearcher.prepareScratchState(HnswGraphSearcher.java:371)
at org.apache.lucene.util.hnsw.HnswGraphSearcher.searchLevel(HnswGraphSearcher.java:284)
at org.apache.lucene.util.hnsw.AbstractHnswGraphSearcher.search(AbstractHnswGraphSearcher.java:82)
at org.apache.lucene.util.hnsw.HnswGraphSearcher.search(HnswGraphSearcher.java:33)
at org.apache.lucene.util.hnsw.HnswGraphSearcher.search(HnswGraphSearcher.java:156)
at org.apache.lucene.util.hnsw.HnswGraphSearcher.search(HnswGraphSearcher.java:98)
at org.apache.lucene.util.hnsw.TestHnswSearchUnknownMaxConn.testSearchGraphWithUnknownMaxConn(TestHnswSearchUnknownMaxConn.java:82)
# gradle, main 5adff78bb168, seed DEADBEEF, assertions on; the same with -Ptests.asserts=false
# release jars, assertions off: 10.2.2 passes; 10.3.0 fails at searchLevel(HnswGraphSearcher.java:283); 10.6.0 at prepareScratchState(:377) <- searchLevel(:290)- The dispatch checks for the constant explicitly ("We can only use filtered search if we know the maxConn") and keeps such graphs on the plain searcher.
- Only
FilteredHnswGraphSearcherasserts a known maxConn; nothing documents that limit for the plain searcher. - 10.2.x searched such graphs fine; the sizing came with bulk scoring in 10.3.0 (#14978), never discussing an unknown maxConn.
The full argument194 words
- Checked explicitly. The dispatch names the constant and excludes it only from filtered search, so such graphs stay on the plain searcher (
HnswGraphSearcher.java:133-144); "graph must have known max connections" is asserted only inFilteredHnswGraphSearcher(:51). - No restriction anywhere.
HnswGraph,HnswGraphSearcherand its public staticsearchmethods carry no@lucene.internalor@lucene.experimentaltag (only the package-private baseAbstractHnswGraphSearcheris experimental). No javadoc, CHANGES or MIGRATE entry says the plain searcher needs a known maxConn. - It worked before. #14160 (
0614fe9ebd7, 10.2.0) added the constant, madeHnswGraph.EMPTYreport it and added the dispatch guard; in 10.2.x the plain searcher readmaxConn()only in that guard. The same commit made a test graph,TestHnswUtil.MockGraph, return the constant. #14978 (10.3.0) added themaxConn() * 2sizing and #15500 (10.4.0) widened it; neither review thread mentions an unknown maxConn. - Incidental failure. A
NegativeArraySizeExceptionfrom an array allocation, not a validation with a message. - Not claimed. Searching
HnswGraph.EMPTYdirectly also throws, but searching an empty graph is meaningless and no reader does it. A graph that reports a maxConn smaller than half its real degree can overflow the same scratch; that graph misreports M.
The devil's-advocate check118 words
maxConn() is documented as "M", the constant has no javadoc and arrived for empty and legacy graphs, no Lucene graph reaches the path, and the failure is loud with an easy workaround, so "report a bound" may be the expected answer. This doc's cross-model fact check rated that case moderate: a maintainer could reasonably close the issue as "report a real M". Why it does not settle it: the dispatch excludes the constant only from filtered search, the plain path worked for it until 10.2.x, and nothing documents the new requirement. These points lower the priority, not the reality. Whether the fix makes it work again or rejects the constant with a clear error is the maintainers' call.
Recommended A. Clamp the initial size at 0 and grow while filling (recommended)
In prepareScratchState, clamp the initial scratch size at 0, and where the scratch is filled, grow bulkNodes and bulkScores when the next neighbour does not fit. The repro passes with assertions on and off, and 218 HNSW and kNN query tests pass.
private void prepareScratchState(int capacity, int bulkScoreSize) {
...
+ // graph.maxConn() may be HnswGraph.UNKNOWN_MAX_CONN (-1): start empty and grow while filling
+ bulkScoreSize = Math.max(0, bulkScoreSize);
if (bulkNodes == null || bulkNodes.length < bulkScoreSize) {
// at both fill sites (findBestEntryPoint and searchLevel):
+ if (numNodes == bulkNodes.length) {
+ growBulkScratch(numNodes + 1); // ArrayUtil.grow on bulkNodes, growExact on bulkScores
+ }
bulkNodes[numNodes++] = friendOrd;All options and trade-offs3 options ยท 512 words
Three directions; the choice is the maintainers'. A restores the 10.2.x behaviour, B makes the requirement explicit, C avoids the per-neighbour check but relies on an undocumented method. A is recommended because it restores what 10.2.x did and is the validated option; this doc's cross-model fact check called B the more defensible default for a maintainer and A the more compatible one.
In prepareScratchState clamp bulkScoreSize at 0; at the two fill sites (findBestEntryPoint, searchLevel) grow bulkNodes with ArrayUtil.grow and bulkScores with ArrayUtil.growExact when the next neighbour does not fit. 16 added lines in HnswGraphSearcher.java.
private void prepareScratchState(int capacity, int bulkScoreSize) {
...
+ // graph.maxConn() may be HnswGraph.UNKNOWN_MAX_CONN (-1): start empty and grow while filling
+ bulkScoreSize = Math.max(0, bulkScoreSize);
if (bulkNodes == null || bulkNodes.length < bulkScoreSize) {
// at both fill sites (findBestEntryPoint and searchLevel):
+ if (numNodes == bulkNodes.length) {
+ growBulkScratch(numNodes + 1); // ArrayUtil.grow on bulkNodes, growExact on bulkScores
+ }
bulkNodes[numNodes++] = friendOrd;Trade-offs
Restores the 10.2.x behaviour for the documented constant and keeps the dispatch as it is. It adds one compare per neighbour in the two inner loops; not benchmarked, and SECURITY.md asks that robustness changes not slow hot paths, so a quick benchmark may be wanted. It also changes behaviour on purpose for a graph that reports a real maxConn but has a node with more than 2 * maxConn() new neighbours: that graph no longer throws ArrayIndexOutOfBoundsException in the plain searcher. That is a hardening choice for the maintainers to accept or drop (read, not claimed).
What the checks validated
- Gradle in a scratch worktree at
5adff78bb168, seedDEADBEEF: the repro fails without the fix (assertions on and off) and passes with it (both). - With the fix, assertions on: every
org.apache.lucene.util.hnswtest plusTestKnnFloatVectorQueryandTestKnnByteVectorQuerypass (14 classes, 218 tests, 0 failures, 3 skipped), and:lucene:core:checkGoogleJavaFormatpasses. - The grow path, which the one-node repro does not reach, was exercised by a probe on the same build: a two-node graph with one and with two levels, and with
acceptOrds, returns the better node with the fix and throws without it.
Throw IllegalArgumentException at the top of HnswGraphSearcher.search when graph.maxConn() is negative (the constant is -1; testing < 1 would also reject 0, which a graph with no links can report and today's plain path handles), drop the special case in the dispatch, and document that search needs a real M.
Trade-offs
No hot-path cost and a clear message, but it turns the 10.2.x behaviour into a documented error; callers then have to report a bound. Not run.
After graphSeek, make sure the scratch holds neighborCount() entries.
Trade-offs
One check per expanded node instead of per neighbour, but neighborCount() has no javadoc, and the on-heap searcher's graphSeek does not move the graph's own cursor, so it is less safe. Not run.
Graph adapters that search another library's HNSW graph with Lucene's HnswGraphSearcher need to report a real maxConn() (the graph's M, or at least half the longest neighbour list, rounded up) on 10.3.0 and later until this is fixed.
A proof-obligation audit of HnswGraphSearcher's contract: because the dispatch sends graphs with UNKNOWN_MAX_CONN to the plain searcher, that searcher must not use maxConn() as a size. The obligation fails at maxConn() * 2; a one-method JUnit test on the public API witnessed it.
The method in full74 words
The obligation: the dispatch (HnswGraphSearcher.java:133-137) sends a graph that reports UNKNOWN_MAX_CONN to the plain searcher, so every use of maxConn() on that path must accept -1. The guard discharges it for the filtered searcher; the sizing at :223 and :284 breaks it for the plain one. A one-method JUnit test on the public API then witnessed the failure on main, and runs on release jars placed the regression between 10.2.2 and 10.3.0.
No prior report found in apache/lucene issues, PRs and commits (incl. the JIRA archive), the mailing lists, downstream trackers or the web (searched 2026-11-03; GitHub re-checked the same day). Nearest: #14160, #14978 and #15500, the PRs that added the constant and the sizing.
Searches5 searches
Exact identifiers (UNKNOWN_MAX_CONN, bulkNodes, NegativeArraySizeException with HnswGraphSearcher) return nothing. Downstream context, not a report: OpenSearch k-NN's adapter over a Faiss graph returned UNKNOWN_MAX_CONN for a few days under Lucene 10.2.1 (k-NN#2721), switched to a real value before 10.3.0 shipped (k-NN#2677), and in February 2026 changed it to M because the searcher allocates twice maxConn (k-NN#3117, about heap use); no failure was reported.
| Where | When | How | Found |
|---|---|---|---|
| GitHub issues, PRs and commits (apache/lucene, incl. the migrated JIRA archive) | 2026-11-03 | three scripted prior-report searches (141 queries) and manual searches: UNKNOWN_MAX_CONN, "unknown maxConn", "maxConn() * 2", NegativeArraySizeException with HNSW, vector, HnswGraphSearcher or prepareScratchState, bulkNodes, HnswGraph.EMPTY | no report; prepareScratchState only in older unrelated items (#14763, #12820, #11906, #992) |
| Introducing PRs (read) | 2026-11-03 | full comment and review threads of #14978, #15500 and #14160 | no mention of an unknown maxConn |
| Open and recently updated PRs | 2026-11-03 | changed files of 422 apache/lucene PRs | #16815 adds an off-heap searcher sized from a real maxConn and leaves this path unchanged; no other PR touches the sizing |
| Downstream and GitHub-wide | 2026-11-03 | Elasticsearch, OpenSearch, OpenSearch k-NN, Solr, GitHub-wide issue and code search | no report; k-NN#2721, #2677, #3117 are context (see above) |
| Mailing lists and web | 2026-11-03 | dev@, java-user@, issues@, commits@ (lists.apache.org) and 5 web queries | no report; the commits@ hits are the #14160 commit |
Introduced in 10.3.0 by #14978 (71d4ad6780e, bulk scoring), which sized the scratch as maxConn() * 2; #15500 (10.4.0) moved the sizing and added it to the entry-point phase. 10.2.0 and 10.2.2 pass; 10.3.0, 10.4.0, 10.5.2, 10.6.0 and main fail. Not fixed upstream.
Versions tested6 rows
Introduced by #14978 "Add a bulk scoring interface to RandomVectorScorer" (mccullocht; 71d4ad6780e on main, merged 2025-07-31; branch_10x cherry-pick cac1a31b6a3, in 10.3.0 and not in 10.2.2). #15500 (benwtrent; 8c1461e536d, 2025-12-15, 10.4.0) moved the sizing into prepareScratchState and added it to findBestEntryPoint, so git blame on main points there, but 10.3.0 already fails. The constant and the dispatch guard came first, with #14160 (0614fe9ebd7, 2025-02-13, 10.2.0). git log -S 'maxConn() * 2' finds only #14978 and #15500. Commit-level bisection was not run; the 10.2.2 and 10.3.0 runs bracket the one change that reads maxConn() in the searcher.
| Version or commit | Verdict | Result |
|---|---|---|
main 5adff78bb168 | affected | fails at prepareScratchState(:371) <- searchLevel(:284), assertions on and off |
| 10.6.0 | affected | fails at prepareScratchState(:377) <- searchLevel(:290) |
| 10.4.0, 10.5.2 | affected | fail (earlier version runs) |
| 10.3.0 | first affected | fails at searchLevel(HnswGraphSearcher.java:283), where the allocation is inline |
| 10.2.0, 10.2.2 | not affected | pass: the plain searcher reads maxConn() only in the dispatch guard |
| 10.1.0, 9.12.3 | API absent | the test does not compile: no maxConn() or UNKNOWN_MAX_CONN |
An independent reproducer, two executed cross-model reviews (OpenAI GPT-5.6: real, high), a devil's-advocate verifier and an adjudication confirmed it. This doc's gradle runs reproduced it on main and validated the fix.
Checks and verdicts239 words
The cross-model reviews ran on main ea57360912ed, whose HnswGraph.java is identical to 5adff78bb168's and whose HnswGraphSearcher.java differs only in the visited-set change of #16806, which touches no sizing line; the intended-lens test was re-run on 5adff78bb168 and fails there too.
| Check | Verdict | What it found |
|---|---|---|
| finder | reproduced | Two-node test fails on main (assertions on and off), 10.6.0 and 10.3.0; passes on 10.2.2 and with fix A. |
| independent reproducer | real | Own three-node test in a package outside Lucene's fails on main, 10.6.0, 10.5.2 and 10.3.0, passes on 10.2.2; its maxConn = 2 control passes. |
| cross-model review, refute lens (OpenAI GPT-5.6) | real, high | Own four-node graph fails in searchLevel (one level) and findBestEntryPoint (two levels); controls pass; an adapted fix A passes its tests and the upstream HNSW tests. |
| cross-model review, intended lens (OpenAI GPT-5.6) | real, high | Own three-node test fails on main and 10.3.0, passes on 10.2.2; the "legacy metadata, advanced API" reading rated weak. |
| devil's-advocate verifier | real, Low | Built the strongest intended-behaviour case and rebutted it; own star-graph test fails on 10.3.0 and later, an honest-bound control passes everywhere. |
| adjudication, separate re-runs and final repro check | confirmed, high, Low | Fresh-seed re-runs on main fail with assertions on and off and pass with fix A. |
| this doc | reproduced; fix A validated | Gradle on main; release jars 10.2.2 (pass), 10.3.0 and 10.6.0 (fail); known-maxConn control passes. |
The diagram as a still image, generated with AI and checked against the code. It is under 10 MB, the limit for a GitHub attachment.
- PNG 451 KB
Finding record, Lucene and upstream sources, repro files and diagram notes2,272 words
- Registry title
- HnswGraphSearcher.search throws NegativeArraySizeException (-2) on every search of a non-empty HnswGraph whose maxConn() is the public constant HnswGraph.UNKNOWN_MAX_CONN, a value its own dispatch deliberately sends to the unfiltered searcher, because that searcher sizes its bulk-scoring scratch as graph.maxConn() * 2 (since 10.3.0)
- Area
- core / util/hnsw: HnswGraphSearcher (bulk-scoring scratch of the unfiltered searcher)
- Priority
- Low, rank 696: HnswGraphSearcher.search throws NegativeArraySizeException on every search of a custom HnswGraph whose maxConn() is the public UNKNOWN_MAX_CONN constant, in 10.3.0 through 10.6.0 (10.2.x worked). Lucene's own formats always report their M, so only custom graph implementations hit it, loudly, on the first search.
- Impact class
- minor
- Impact note
- Only code that implements its own HnswGraph whose maxConn() returns HnswGraph.UNKNOWN_MAX_CONN (for example a custom vectors format or a graph adapter that does not know M) and searches it with the public static HnswGraphSearcher.search is hit, on every search of a non-empty graph, with -ea and -da, in 10.3.0 through 10.6.0 and on main (10.2.x worked); it fails loudly with NegativeArraySizeException and returns no wrong result; no in-tree graph reaches the path (HnswGraph.EMPTY is never searched, the Lucene90 backward-codec graph uses its own search, every current format reports its stored M), so indexing, merging and kNN queries on Lucene's own formats are not affected.
- Summary
- You implement HnswGraph for a graph whose M you do not know, so maxConn() returns the public constant HnswGraph.UNKNOWN_MAX_CONN (-1), and search it with HnswGraphSearcher.search, whose dispatch handles that value on purpose by skipping only the filtered searcher ('We can only use filtered search if we know the maxConn'). You expect the nearest neighbours, as the same graph gives when it reports a known maxConn and as 10.2.x gives. Instead every search throws java.lang.NegativeArraySizeException: -2 from HnswGraphSearcher.prepareScratchState, because since 10.3.0 the unfiltered searcher allocates its bulk-scoring scratch as new int[graph.maxConn() * 2] (in searchLevel, and since 10.4.0 also in findBestEntryPoint for graphs with more than one level).
- Introduced
- 71d4ad6780e58a0768cb33e5f17320977bbc015d (apache/lucene#14978 'Add a bulk scoring interface to RandomVectorScorer', by mccullocht; released through the branch_10x cherry-pick cac1a31b6a32d09c57f06d41d7d9d5802274828e, contained in releases/lucene/10.3.0 and not in 10.2.2): it added bulkNodes = new int[graph.maxConn() * 2] (and bulkScores) to HnswGraphSearcher.searchLevel, which is new int[-2] for UNKNOWN_MAX_CONN. #15500 (8c1461e536d, by benwtrent, 10.4.0; branch_10x 3c2a0f41a51) later moved the sizing into prepareScratchState(size, graph.maxConn() * 2) and added the same call to findBestEntryPoint, so git blame on main's current lines points to #15500, but 10.3.0 already fails. The constant and the dispatch guard that sends such graphs to the plain searcher came first, with #14160 (0614fe9ebd7, by benwtrent, 10.2.0), when that searcher never read maxConn() [read: git log -S 'maxConn() * 2' and git log -S UNKNOWN_MAX_CONN in the full history clone each find only these commits; git show on the release tags; exec: 10.2.2 passes, 10.3.0 fails; commit-level bisection not run, the two release runs bracket the single relevant commit] (2025-07-31 (#14978: author date 2025-07-31T04:01:34-07:00, merged 2025-07-31T11:01:34Z; branch_10x commit 2025-08-06); the findBestEntryPoint call of #15500 dates from 2025-12-15 (author date 2025-12-15T11:22:48-05:00); #14160, which added the constant and the fallback, from 2025-02-13)
- First affected version
- 10.3.0 [exec: the test fails on 10.3.0 with NegativeArraySizeException -2 at HnswGraphSearcher.searchLevel(HnswGraphSearcher.java:283), where the allocation is inline, and passes on 10.2.0 and 10.2.2]; HnswGraph.maxConn() and UNKNOWN_MAX_CONN first exist in 10.2.0 (#14160), so 10.1.x, 10.0.x and 9.x cannot express the trigger (the test does not compile on 10.1.0 or 9.12.3)
- Affected versions
- 10.3.0 through 10.6.0 (released 2026-10-29, 4ae946e5a9b) and main 5adff78bb168 (12.0.0-SNAPSHOT); never fixed. [exec, JDK 25.0.3, Linux aarch64] main (-ea and -da: finder, the independent reproduction, the intended-lens review's re-check, the intended-behaviour check, adjudication, registration check), 10.6.0 (-da: finder, the independent reproduction, the intended-behaviour check, history step), 10.5.2 (-da: the independent reproduction), 10.4.0 (-da: history step) and 10.3.0 (-da: finder, the independent reproduction, the cross-model reviews, the intended-behaviour check, history step) all fail with NegativeArraySizeException: -2 (at prepareScratchState :371 <- searchLevel :284 on main, :377 <- :290 on 10.4.0 to 10.6.0, searchLevel :283 on 10.3.0); 10.2.0 and 10.2.2 pass (finder, the independent reproduction, the cross-model reviews, the intended-behaviour check, history step); on 10.1.0 and 9.12.3 the test does not compile because HnswGraph has no maxConn() or UNKNOWN_MAX_CONN there (history step). [read] the sizing 'maxConn() * 2' is absent from HnswGraphSearcher.java at releases/lucene/10.2.0, 10.2.1 and 10.2.2 and present at every tag from 10.3.0 to 10.6.0 (10.3.0-10.3.2 inline in searchLevel; 10.4.0-10.6.0 in prepareScratchState calls at :229 and :290). Not new in 10.6 (10.5.2 and older releases fail). Not fixed upstream: upstream main head still has prepareScratchState(size, graph.maxConn() * 2) at HnswGraphSearcher.java:223 and :284 and new int[bulkScoreSize] at :371; the newest commit on the file is #16806 (2e21f30ef68, author date 2026-11-02), which 5adff78bb168 contains [read: GitHub API, 2026-11-03].
- Status
- documented (brief, checks and a per-finding doc exist)
- Verification
- Confirmed by AI-run checks: an independent repro, a cross-model review that agreed, an intended-behaviour check, a prior-report search and a skeptical audit; not yet triaged by a person.
- Found by
- semi-formal: proof-obligation audit
- Upstream and prior reports
- none found (2026-11-03, on top of the finder's run and the independent reproduction's, the intended-behaviour check's and the adjudication's searches; rechecked the same day): three scripted prior-report searches (the finder's 51 queries with 3474 distinct hits, 7 high; two follow-up runs of 45 queries each with 112 and 126 distinct hits, 1 high; none a report) over GitHub issues and PRs, commits, downstream repositories, the dev@, java-user@, issues@ and commits@ lists, the JIRA archive and the web; 24 manual GitHub issue and PR searches (UNKNOWN_MAX_CONN, 'unknown maxConn', 'maxConn() * 2', NegativeArraySizeException with HNSW, vector, HnswGraphSearcher or prepareScratchState, bulkNodes, prepareScratchState, HnswGraph.EMPTY, custom HnswGraph implementations) on apache/lucene, GitHub-wide, Elasticsearch, OpenSearch, OpenSearch k-NN and Solr, plus 4 code searches; the changed files of 422 open or recently updated apache/lucene PRs; the full comment and review threads of #14978, #15500, #14160 and six related items; commit author dates on the searcher files; lists.apache.org searches and 5 web queries: no report of the failure or of the unknown-maxConn scratch sizing. Re-check: UNKNOWN_MAX_CONN, 'unknown maxConn', 'NegativeArraySizeException HnswGraphSearcher' and bulkNodes return nothing (apache/lucene and GitHub-wide), prepareScratchState only the older #14763, #12820, #11906 and #992; the only PR updated since 2026-11-02 that touches the hnsw searcher files is #16806 (merged, in 5adff78bb168); upstream main head still has the sizing at HnswGraphSearcher.java:223 and :284. Related history, not reports: #14160 (added the constant and the unfiltered fallback), #14978 and #15500 (added the sizing), open PR #16815 (adds an off-heap graph searcher sized from a real maxConn and leaves this path unchanged), and OpenSearch k-NN #2721 and #2677 (a downstream graph adapter searched with HnswGraphSearcher returned UNKNOWN_MAX_CONN for four days under Lucene 10.2.1, then a real value before 10.3.0, with no failure reported).
- Lucene commit tested
- main
5adff78bb168(12.0.0-SNAPSHOT): gradle in a scratch worktree for this doc (the repro fails with assertions on and off and passes with fix A; with it 218 HNSW and kNN query tests pass and the formatter check passes) and standalone runs of the repro (a known-maxConn control passes); release jars 10.2.2 (pass), 10.3.0 and 10.6.0 (fail) with this repro. Earlier version runs: 10.2.0 passes, 10.4.0 and 10.5.2 fail. The cross-model reviews ran on mainea57360912ed, whoseHnswGraph.javais identical and whoseHnswGraphSearcher.javadiffers only in an unrelated visited-set change (#16806).
- Finding record
lucene/core/src/java/org/apache/lucene/util/hnsw/HnswGraphSearcher.java:89-99 (at 5adff78bb168) (public static search(scorer, collector, graph, acceptOrds)), :114-157 (search with filteredDocCount; :133-137 'We can only use filtered search if we know the maxConn' excludes UNKNOWN_MAX_CONN from FilteredHnswGraphSearcher only; :141-144 builds the plain HnswGraphSearcher), :216-223 (findBestEntryPoint: returns early for entryNode -1 or one level, else prepareScratchState(size, graph.maxConn() * 2) at :223), :282-288 (searchLevel: prepareScratchState(size, graph.maxConn() * 2) at :284, before any early exit), :363-374 (prepareScratchState: bulkNodes = new int[bulkScoreSize] at :371, the throw; fix site)lucene/core/src/java/org/apache/lucene/util/hnsw/HnswGraph.java:53-57 (at 5adff78bb168) (public abstract class, public static final int UNKNOWN_MAX_CONN = -1, protected sole constructor), :88-89 (maxConn() 'returns M, the maximum number of connections for a node', no lower bound), :137-140 (HnswGraph.EMPTY returns UNKNOWN_MAX_CONN); AbstractHnswGraphSearcher.java:26 (@lucene.experimental on the base class only), :74-83 (search: findBestEntryPoint, UNK_EP early return, then searchLevel at :82); FilteredHnswGraphSearcher.java:51 (assert graph.maxConn() > 0 : 'graph must have known max connections'); lucene/core/src/java/module-info.java:50 (exports org.apache.lucene.util.hnsw)lucene/backward-codecs/src/java/org/apache/lucene/backward_codecs/lucene90/Lucene90HnswVectorsReader.java:498 (at 5adff78bb168) (the only other non-test UNKNOWN_MAX_CONN source; that graph is searched through Lucene90OnHeapHnswGraph.search); lucene/CHANGES.txt (GITHUB#14978 in the 10.3.0 section, GITHUB#15500 in the 10.4.0 section)Two Lucene main checkouts: git diff ea57360912ed 5adff78bb168 is empty for HnswGraph.java; for HnswGraphSearcher.java it is the visited-set change of #16806 and touches no maxConn, bulk-scoring or prepareScratchState line (2026-11-03)https://github.com/apache/lucene/pull/14978, https://github.com/apache/lucene/pull/15500, https://github.com/apache/lucene/pull/14160, https://github.com/apache/lucene/pull/16815, https://github.com/apache/lucene/pull/16806, https://github.com/opensearch-project/k-NN/pull/2721, https://github.com/opensearch-project/k-NN/pull/2677, read 2026-11-03
- The finding in 30 seconds
lucene/core/src/java/org/apache/lucene/util/hnsw/HnswGraph.java:53-57 (at 5adff78bb168) (public abstract class, public static final int UNKNOWN_MAX_CONN = -1, protected constructor), :88-89 (maxConn() 'returns M, the maximum number of connections for a node'), :137-140 (HnswGraph.EMPTY returns UNKNOWN_MAX_CONN); lucene/core/src/java/module-info.java:50 (exports org.apache.lucene.util.hnsw)lucene/core/src/java/org/apache/lucene/util/hnsw/HnswGraphSearcher.java:131-145 (at 5adff78bb168) (dispatch: :133-137 'We can only use filtered search if we know the maxConn' excludes UNKNOWN_MAX_CONN from FilteredHnswGraphSearcher only; :141-144 builds the plain HnswGraphSearcher); :89-99 (public static search(scorer, collector, graph, acceptOrds))lucene/core/src/java/org/apache/lucene/util/hnsw/HnswGraphSearcher.java:216-223 (at 5adff78bb168) (findBestEntryPoint: early return for entryNode -1 or one level, else prepareScratchState(size, graph.maxConn() * 2) at :223), :282-288 (searchLevel: prepareScratchState(size, graph.maxConn() * 2) at :284, before any early exit), :363-374 (prepareScratchState: bulkNodes = new int[bulkScoreSize] at :371), :42-43 (bulkNodes, bulkScores start null), :244 and :330 (the two fill sites)lucene/core/src/java/org/apache/lucene/util/hnsw/AbstractHnswGraphSearcher.java:26 (at 5adff78bb168) (@lucene.experimental, package-private base class), :74-83 (search: findBestEntryPoint, then searchLevel at :82); FilteredHnswGraphSearcher.java:51 (assert graph.maxConn() > 0 : 'graph must have known max connections'), :52-54 (exploration multiplier from maxConn)lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99HnswVectorsReader.java:368, (at 5adff78bb168) :376-378 (no HNSW search when the graph is empty, so HnswGraph.EMPTY is never searched), :652-653 (OffHeapHnswGraph.maxConn returns the stored M); lucene/core/src/java/org/apache/lucene/util/hnsw/OnHeapHnswGraph.java:205-206; lucene/backward-codecs/src/java/org/apache/lucene/backward_codecs/lucene90/Lucene90HnswVectorsReader.java:497-498 (returns UNKNOWN_MAX_CONN), :264 (searched by Lucene90OnHeapHnswGraph.search, not HnswGraphSearcher)lucene/core/src/test/org/apache/lucene/util/hnsw/TestHnswUtil.java:201 (at 5adff78bb168) (MockGraph), :277-279 (maxConn() returns UNKNOWN_MAX_CONN; used only by HnswUtil tests, no HnswGraphSearcher in the file); HnswGraphSearcher.java:218-220 and AbstractHnswGraphSearcher.java:77-81 (no entry node: return before the sizing)
- Who is affected
lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99HnswVectorsReader.java:368, (at 5adff78bb168) :376-378 (no HNSW search when the graph is empty, so HnswGraph.EMPTY is never searched), :652-653 (OffHeapHnswGraph.maxConn returns the stored M); lucene/core/src/java/org/apache/lucene/util/hnsw/OnHeapHnswGraph.java:205-206; lucene/backward-codecs/src/java/org/apache/lucene/backward_codecs/lucene90/Lucene90HnswVectorsReader.java:497-498 (returns UNKNOWN_MAX_CONN), :264 (searched by Lucene90OnHeapHnswGraph.search, not HnswGraphSearcher)lucene/core/src/java/org/apache/lucene/util/hnsw/AbstractHnswGraphSearcher.java:26 (at 5adff78bb168) (@lucene.experimental, package-private base class), :74-83 (search: findBestEntryPoint, then searchLevel at :82); FilteredHnswGraphSearcher.java:51 (assert graph.maxConn() > 0 : 'graph must have known max connections'), :52-54 (exploration multiplier from maxConn)https://github.com/opensearch-project/k-NN/pull/2721, https://github.com/opensearch-project/k-NN/pull/2677, https://github.com/opensearch-project/k-NN/pull/3117 (downstream context, read 2026-11-03)lucene/core/src/java/org/apache/lucene/util/hnsw/HnswGraphBuilder.java:131-132 (at 5adff78bb168) (javadoc: M connections on upper layers, M * 2 on the lowest level), :424 (maxConnOnLevel = level == 0 ? M * 2 : M); HnswGraphSearcher.java:133-139 (filtered searcher only when the strategy picks it)
- Why it looks unintended
lucene/core/src/java/org/apache/lucene/util/hnsw/HnswGraphSearcher.java:131-145 (at 5adff78bb168) (dispatch: :133-137 'We can only use filtered search if we know the maxConn' excludes UNKNOWN_MAX_CONN from FilteredHnswGraphSearcher only; :141-144 builds the plain HnswGraphSearcher); :89-99 (public static search(scorer, collector, graph, acceptOrds))lucene/core/src/java/org/apache/lucene/util/hnsw/AbstractHnswGraphSearcher.java:26 (at 5adff78bb168) (@lucene.experimental, package-private base class), :74-83 (search: findBestEntryPoint, then searchLevel at :82); FilteredHnswGraphSearcher.java:51 (assert graph.maxConn() > 0 : 'graph must have known max connections'), :52-54 (exploration multiplier from maxConn)lucene/core/src/java/org/apache/lucene/util/hnsw/HnswGraph.java:53-57 (at 5adff78bb168) (public abstract class, public static final int UNKNOWN_MAX_CONN = -1, protected constructor), :88-89 (maxConn() 'returns M, the maximum number of connections for a node'), :137-140 (HnswGraph.EMPTY returns UNKNOWN_MAX_CONN); lucene/core/src/java/module-info.java:50 (exports org.apache.lucene.util.hnsw)a Lucene clone releases/lucene/10.2.2:lucene/core/src/java/org/apache/lucene/util/hnsw/HnswGraphSearcher.java:111 (the only maxConn() read, in the dispatch guard); releases/lucene/10.3.0 same file :282-284 (bulkNodes = new int[graph.maxConn() * 2]); releases/lucene/10.6.0 :229, :290 (prepareScratchState(size, graph.maxConn() * 2))lucene/CHANGES.txt:1609 (at 5adff78bb168) (GITHUB#14978 in the 10.3.0 section, :1592), :1490 (GITHUB#15500 in the 10.4.0 section, :1335); lucene/MIGRATE.md (no entry on maxConn for search)https://github.com/apache/lucene/pull/14978, https://github.com/apache/lucene/pull/15500, https://github.com/apache/lucene/pull/14160, https://github.com/apache/lucene/pull/16815 (read 2026-11-03)
- Suggested fix
lucene/core/src/java/org/apache/lucene/util/hnsw/HnswGraphSearcher.java:216-223 (at 5adff78bb168) (findBestEntryPoint: early return for entryNode -1 or one level, else prepareScratchState(size, graph.maxConn() * 2) at :223), :282-288 (searchLevel: prepareScratchState(size, graph.maxConn() * 2) at :284, before any early exit), :363-374 (prepareScratchState: bulkNodes = new int[bulkScoreSize] at :371), :42-43 (bulkNodes, bulkScores start null), :244 and :330 (the two fill sites)Lucene main 5adff78bb168 SECURITY.md:46 (robustness improvements must not slow down hot paths)lucene/core/src/java/org/apache/lucene/util/hnsw/HnswGraphSearcher.java:284-287, (at 5adff78bb168) :370-373 (maxConn() = 0: zero-length scratch, bulkScores grown for the entry points; a graph with no links never fills it)
- How it was found
lucene/core/src/java/org/apache/lucene/util/hnsw/HnswGraphSearcher.java:131-145 (at 5adff78bb168) (dispatch: :133-137 'We can only use filtered search if we know the maxConn' excludes UNKNOWN_MAX_CONN from FilteredHnswGraphSearcher only; :141-144 builds the plain HnswGraphSearcher); :89-99 (public static search(scorer, collector, graph, acceptOrds))lucene/core/src/java/org/apache/lucene/util/hnsw/HnswGraphSearcher.java:216-223 (at 5adff78bb168) (findBestEntryPoint: early return for entryNode -1 or one level, else prepareScratchState(size, graph.maxConn() * 2) at :223), :282-288 (searchLevel: prepareScratchState(size, graph.maxConn() * 2) at :284, before any early exit), :363-374 (prepareScratchState: bulkNodes = new int[bulkScoreSize] at :371), :42-43 (bulkNodes, bulkScores start null), :244 and :330 (the two fill sites)
- Prior reports
https://github.com/apache/lucene/pull/14978, https://github.com/apache/lucene/pull/15500, https://github.com/apache/lucene/pull/14160, https://github.com/apache/lucene/pull/16815 (read 2026-11-03)https://github.com/opensearch-project/k-NN/pull/2721, https://github.com/opensearch-project/k-NN/pull/2677, https://github.com/opensearch-project/k-NN/pull/3117 (downstream context, read 2026-11-03)
- History
a Lucene clone releases/lucene/10.2.2:lucene/core/src/java/org/apache/lucene/util/hnsw/HnswGraphSearcher.java:111 (the only maxConn() read, in the dispatch guard); releases/lucene/10.3.0 same file :282-284 (bulkNodes = new int[graph.maxConn() * 2]); releases/lucene/10.6.0 :229, :290 (prepareScratchState(size, graph.maxConn() * 2))lucene/CHANGES.txt:1609 (at 5adff78bb168) (GITHUB#14978 in the 10.3.0 section, :1592), :1490 (GITHUB#15500 in the 10.4.0 section, :1335); lucene/MIGRATE.md (no entry on maxConn for search)
- Diagrams
lucene/core/src/java/org/apache/lucene/util/hnsw/HnswGraphSearcher.java:131-145 (at 5adff78bb168) (dispatch: :133-137 'We can only use filtered search if we know the maxConn' excludes UNKNOWN_MAX_CONN from FilteredHnswGraphSearcher only; :141-144 builds the plain HnswGraphSearcher); :89-99 (public static search(scorer, collector, graph, acceptOrds))lucene/core/src/java/org/apache/lucene/util/hnsw/HnswGraphSearcher.java:216-223 (at 5adff78bb168) (findBestEntryPoint: early return for entryNode -1 or one level, else prepareScratchState(size, graph.maxConn() * 2) at :223), :282-288 (searchLevel: prepareScratchState(size, graph.maxConn() * 2) at :284, before any early exit), :363-374 (prepareScratchState: bulkNodes = new int[bulkScoreSize] at :371), :42-43 (bulkNodes, bulkScores start null), :244 and :330 (the two fill sites)
- TestHnswSearchUnknownMaxConn.java (lucene/core, package org.apache.lucene.util.hnsw, 1 test):
repro/LFV-1238/TestHnswSearchUnknownMaxConn.java(87 lines)
assets/LFV-1238/bug.png(451 KB)
![Diagram of the bug: a custom HnswGraph reports maxConn() = HnswGraph.UNKNOWN_MAX_CONN (-1). HnswGraphSearcher.search skips the filtered searcher, which needs a known maxConn, and runs the plain searcher. That searcher sizes its bulk-scoring scratch as maxConn() * 2 = -2, and new int[-2] throws NegativeArraySizeException: -2 on every search. Lucene 10.2.2 returned the hit; 10.3.0 and later throw.](../assets/LFV-1238/bug.png)