All findings · Lucene finding · LFV-458 · hnsw-merge-ordmap-hashmap-capacity

HNSW merge cannot reuse a graph of more than 805,306,368 vectors

A merge that reuses an HNSW graph of more than 805,306,368 vectors always throws BufferAllocationException: it presizes an IntIntHashMap that cannot hold that many entries. Lucene writes such a segment without error (code reading) but can then never merge it with its graph reused; 9.9.2 handled this size (its own presize failed only from 1,610,612,672 nodes).

Summary

Priority
Low, rank 967A default-codec HNSW merge that reuses a graph of more than 805,306,368 vectors throws on every attempt and, by code reading, closes the writer, so such a segment can never be merged; it is loud, but it needs only about 0.375 x MAX_DOCS vectors in one segment.
Impact
crash
Since
9.10.0 (#12950, 2023-12-15)introduced 2023-12-15 · about 2 years
Affected
9.10.0 to 10.6.0, main and branch_11x, when a merge reuses a graph of more than 805M vectors (10.4.0 and 10.5.0: custom codec only)
Status
documented
Found by
code reading of the HNSW merge path against MAX_DOCS; bound computed, then executed

AI-assisted page: this finding was first signalled by code reading or testing and then checked with AI tools; the checks it went through, and what each found, are under Prior reports, history and checks. Not yet triaged by a human. Snapshot of 2026-11-06.

The finding in 30 seconds

When Lucene merges segments it reuses the biggest HNSW graph instead of rebuilding it. The table it builds to renumber that graph's nodes cannot hold more than 805,306,368 entries.

How a merge reuses an HNSW graph

  1. Each segment has its own HNSW graph. Its nodes are numbered 0, 1, 2, ... in the segment's doc order; these numbers are the ordinals.
  2. A merge gives every document a new doc id in the merged segment. Rebuilding a big graph is slow, so the merge copies the existing graph and only renumbers its nodes.
  3. To renumber, it needs each old node's new ordinal. It builds a lookup table from new doc id to old node, then walks the merged documents in order and looks each one up.
Diagram of the failing merge: a segment's HNSW graph with 805,306,369 vectors; the merge presizes a new-doc-to-old-node hash table to that count; the table has at most 2^30 = 1,073,741,824 slots and keeps a quarter of them free, so it holds at most 805,306,368 entries; its constructor throws BufferAllocationException before the table is allocated.
The failing merge: the reused graph's 805,306,369 nodes, the lookup table presized to that count, and the table's limit of 805,306,368 entries. Select the still to open it at full size. Generated with AI, checked against the code.

The finding, step by step

  1. One segment holds an HNSW vector field with 805,306,369 vectors. By code reading, Lucene writes such a segment without complaint, for example by force-merging smaller ones: nothing below IndexWriter.MAX_DOCS (2,147,483,519) limits it (writing one was not run end to end).
  2. A later merge reuses that segment's graph instead of rebuilding it. To renumber the graph's nodes it first fills a lookup table from new doc id to old node, an IntIntHashMap presized to the graph's 805,306,369 nodes.
  3. You expect the merge to go on, as it does with 805,306,368 vectors and as Lucene 9.9.2 did with 805,306,369.
  4. Instead the map's constructor throws BufferAllocationException: Maximum array size exceeded for this load factor. The map keeps a quarter of its slots free and has at most 2^30 = 1,073,741,824 slots, so it fits at most 805,306,368 entries. It throws before the map's arrays are allocated.
  5. Every merge that reuses the graph fails the same way, so the segment can never be merged with its graph reused. By code reading, inside IndexWriter each attempt first rewrites the segment's other data and flat vectors, and the failure is tragic: the writer rolls back to the last commit and closes.

Who is affected

Only an index with one segment of more than 805 million vectors in one HNSW field (0.375 x MAX_DOCS), far past the default 5 GB merge cap (built by forceMerge, a raised cap, a custom merge policy or addIndexes), when a merge reuses that graph. Default codec, both mergers, 9.10.0 to 10.6.0, main and branch_11x (10.4.0, 10.5.0: custom codec only).

Mechanism, terms and numbers574 words

When a merge reuses an existing segment's HNSW graph, IncrementalHnswGraphMerger.getNewOrdMapping first maps new doc ids to the graph's old ordinals in an IntIntHashMap presized to the graph's node count. Lucene's hash containers default to a 0.75 load factor and cap the table at 2^30 slots, so the constructor throws BufferAllocationException for more than 805,306,368 expected entries, about 0.375 x IndexWriter.MAX_DOCS. No HNSW limit is documented or enforced below MAX_DOCS, so, by code reading, Lucene writes a segment with a bigger graph (for example by merging smaller ones) and then can never merge it again with its graph reused: every attempt throws, and by code reading IndexWriter treats it as a tragic event. The map came from #12950 (9.10.0), a box/unbox speed-up that replaced a java.util.HashMap; 9.9.2 completes the same mapping (its own presize failed only from 1,610,612,672 nodes). The repro drives the merger directly with a stub graph; no real IndexWriter merge of this size was run. The scale is extreme, the failure is loud, and nothing is corrupted.

The mechanism (main 645d4fe14a20)

  1. Default path. The default codec uses Lucene99HnswVectorsFormat (Lucene104Codec.java:123). On a merge, Lucene99HnswVectorsWriter.buildAndWriteGraph calls addReader for every merging segment with the field and then merger.merge(...) (:510-525); createGraphMerger returns IncrementalHnswGraphMerger by default, and ConcurrentHnswMerger when the format has a merge executor or the scheduler supplies one with numMergeWorkers > 1 (:668-688). On main the per-field reader is itself an HnswGraphProvider (PerFieldKnnVectorsFormat.java:198, :357-369), so default-codec graphs are reused.
  2. Which graphs are mapped. addReader (IncrementalHnswGraphMerger.java:110-147) picks as base the graph with the most live vectors among those with at most 40% deletions (:133-139) and keeps every graph without deletions (:142-144). createBuilder calls getNewOrdMapping (:166); ConcurrentHnswMerger calls the same method (ConcurrentHnswMerger.java:79).
  3. The presize. getNewOrdMapping (:204-250) does new IntIntHashMap(graphReaders.get(i).graphSize) (:221) for each reused graph.
  4. The capacity. The constructor calls ensureCapacity, which computes minBufferSize(expectedElements, 0.75) before allocating (IntIntHashMap.java:85-107, :446-455). HashContainers fixes DEFAULT_LOAD_FACTOR = 0.75f (:33) and MAX_HASH_ARRAY_LENGTH = 2^30 (:48); minBufferSize throws when ceil(n / 0.75), rounded up to a power of two, exceeds 2^30 (:81-99). Largest n that fits: 805,306,368.
  5. Presizing less does not help. Growth stops at the same 2^30-slot table (nextBufferSize, HashContainers.java:56-63): 805,306,368 non-zero keys plus the separately stored key 0 (IntIntHashMap.java:120-124, growth only at :139-140). Without the presize one more entry fits at most; from 805,306,370 only another structure fixes it.
  6. In IndexWriter (code reading, not executed). The exception is a RuntimeException, not a MergeAbortedException, so handleMergeException rethrows it (IndexWriter.java:5043-5057), IndexWriter.merge calls tragicEvent(t, "merge") (:5109-5113), and the writer rolls back and closes (:6098-6106). A waiting forceMerge caller sees an IOException whose cause is this exception (:2292-2301).
  7. Where in an IndexWriter merge (code reading). The map is built in the graph phase: SegmentMerger.merge first writes the merged stored fields, norms, postings, doc values and points (SegmentMerger.java:123-158), then the vectors, where KnnVectorsWriter.merge writes the merged flat vectors in phase 1 (KnnVectorsWriter.java:133-150) and builds the graphs in phase 2 (:152-158). So each failing attempt first rewrites all of that (about 412 GB of vectors alone at 128 dimensions). The concurrent merger also allocates a FixedBitSet of maxOrd bits (about 96 MiB) before the map (ConcurrentHnswMerger.java:78).
Exposure and likelihood404 words
  • Versions through IndexWriter. With the default codec 9.10.0-10.3.2, 10.5.1-10.6.0 and main reach the line. 10.4.0 and 10.5.0 checked the per-field wrapper instead of the unwrapped reader, so with the default codec "every merge rebuilt the graph from scratch" (CHANGES entry of #16400, fixed by #16403 in 10.5.1; the issue itself is still open); there it needs a codec whose vectors format is not wrapped per field. Up to 10.3.2 only graphs without deletions are reused; since 10.4.0 the base may have up to 40% deletions. branch_11x has main's single site; branch_10_6 keeps 10.6.0's two sites (IncrementalHnswGraphMerger.java:210, ConcurrentHnswMerger.java:147).
  • How such a segment arises. Not by flushing, and not by default background merges: TieredMergePolicy caps them at 5 GB, and 805M vectors are about 412 GB of float32 data at 128 dimensions alone. forceMerge(n) overrides the cap, as do a raised maxMergedSegmentMB, a custom merge policy and addIndexes(CodecReader...), which merges the readers it is given; addIndexes(Directory...) copies a giant segment built elsewhere as it is. Two graphs of 500M and 400M vectors each map fine, so by code reading Lucene writes the 900M-vector segment (no count guard found; not run end to end); the next merge that reuses that graph fails.
  • What it costs. Every merge that reuses that graph throws, e.g. forceMerge(1) after new documents arrive (a forceMerge with nothing to merge does nothing). From 10.4.0 (10.5.1 with the default codec) a background merge can hit it too: TieredMergePolicy considers a too-large segment once both its own and the index-wide delete percentages pass the default 20% deletesPctAllowed, which may get it selected, and the graph is reused while at most 40% of it is deleted. On 9.10.0-10.3.2 a graph with deletions is not reused, so such merges rebuild it and succeed. A failing merge, by code reading, closes the writer, losing uncommitted changes, and fails again whenever a later merge reuses that graph; inside IndexWriter each attempt first rewrites the segment's other data and flat vectors before it throws.
  • Scale. Building or merging such a graph needs a very large heap: estimates range from 50-75 GB to 200-300 GB (not measured); the executed 9.9.2 mapping at this size used about 70 GiB. Open work on merge graphs larger than the heap (#16815, #15904) would make the bound easier to reach.
  • Loud, not silent. No wrong results and no corruption; the committed index stays as it was (code reading).

Minimal repro

One JUnit test in org.apache.lucene.util.hnsw, public merger API only: one segment whose stub graph reports 805,306,369 nodes and whose vectors share one array, so nothing large is allocated. A correct merger maps the ordinals and starts copying the graph, where the stub throws the test's marker; on main the map's constructor throws first.

TestHnswMergeLargeGraphOrdMapping.java (JUnit, one test)169 lines
/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package org.apache.lucene.util.hnsw;

import java.util.Collections;
import java.util.Map;
import org.apache.lucene.codecs.KnnVectorsReader;
import org.apache.lucene.codecs.hnsw.DefaultFlatVectorScorer;
import org.apache.lucene.codecs.hnsw.HnswGraphProvider;
import org.apache.lucene.index.ByteVectorValues;
import org.apache.lucene.index.DocValuesSkipIndexType;
import org.apache.lucene.index.DocValuesType;
import org.apache.lucene.index.FieldInfo;
import org.apache.lucene.index.Float16VectorValues;
import org.apache.lucene.index.FloatVectorValues;
import org.apache.lucene.index.IndexOptions;
import org.apache.lucene.index.MergePolicy;
import org.apache.lucene.index.VectorEncoding;
import org.apache.lucene.index.VectorSimilarityFunction;
import org.apache.lucene.search.AcceptDocs;
import org.apache.lucene.search.KnnCollector;
import org.apache.lucene.tests.util.LuceneTestCase;
import org.apache.lucene.util.InfoStream;

public class TestHnswMergeLargeGraphOrdMapping extends LuceneTestCase {

  private static final int N = 805_306_369;

  public void testMergeReusesGraphOfMoreThan805MillionVectors() throws Exception {
    FieldInfo fieldInfo =
        new FieldInfo(
            "v",
            0,
            false,
            false,
            false,
            IndexOptions.NONE,
            DocValuesType.NONE,
            DocValuesSkipIndexType.NONE,
            -1,
            Map.of(),
            0,
            0,
            0,
            2,
            VectorEncoding.FLOAT32,
            VectorSimilarityFunction.EUCLIDEAN,
            false,
            false);
    FloatVectorValues vectors =
        FloatVectorValues.fromFloats(Collections.nCopies(N, new float[2]), 2);
    IncrementalHnswGraphMerger merger =
        new IncrementalHnswGraphMerger(
            fieldInfo,
            DefaultFlatVectorScorer.INSTANCE.getRandomVectorScorerSupplier(
                VectorSimilarityFunction.EUCLIDEAN, vectors),
            16,
            100);
    merger.addReader(new GraphSegmentReader(vectors), doc -> doc, null);
    // expected: ordinals are mapped and the graph copy starts; actual: BufferAllocationException
    expectThrows(GraphCopyStarted.class, () -> merger.merge(vectors, InfoStream.NO_OUTPUT, N));
  }

  private static final class GraphCopyStarted extends RuntimeException {}

  private static final class GraphSegmentReader extends KnnVectorsReader
      implements HnswGraphProvider {
    private final FloatVectorValues vectors;

    GraphSegmentReader(FloatVectorValues vectors) {
      this.vectors = vectors;
    }

    @Override
    public HnswGraph getGraph(String field) {
      return new HnswGraph() {
        @Override
        public int size() {
          return N;
        }

        @Override
        public int maxConn() {
          throw new GraphCopyStarted();
        }

        @Override
        public int numLevels() {
          throw new GraphCopyStarted();
        }

        @Override
        public int entryNode() {
          throw new GraphCopyStarted();
        }

        @Override
        public NodesIterator getNodesOnLevel(int level) {
          throw new GraphCopyStarted();
        }

        @Override
        public void seek(int level, int target) {
          throw new GraphCopyStarted();
        }

        @Override
        public int nextNeighbor() {
          throw new GraphCopyStarted();
        }

        @Override
        public int neighborCount() {
          throw new GraphCopyStarted();
        }
      };
    }

    @Override
    public FloatVectorValues getFloatVectorValues(String field) {
      return vectors;
    }

    @Override
    public ByteVectorValues getByteVectorValues(String field) {
      throw new UnsupportedOperationException();
    }

    @Override
    public Float16VectorValues getFloat16VectorValues(String field) {
      throw new UnsupportedOperationException();
    }

    @Override
    public void search(String field, float[] target, KnnCollector collector, AcceptDocs docs) {
      throw new UnsupportedOperationException();
    }

    @Override
    public void search(String field, byte[] target, KnnCollector collector, AcceptDocs docs) {
      throw new UnsupportedOperationException();
    }

    @Override
    public void search(String field, short[] target, KnnCollector collector, AcceptDocs docs) {
      throw new UnsupportedOperationException();
    }

    @Override
    public void checkIntegrity(MergePolicy.OneMerge merge) {}

    @Override
    public void close() {}
  }
}
Command
./gradlew :lucene:core:test --tests org.apache.lucene.util.hnsw.TestHnswMergeLargeGraphOrdMapping -Ptests.seed=DEADBEEF
Expected
The merger maps all 805,306,369 ordinals and starts the graph copy (the test's GraphCopyStarted), as it does with 805,306,368 vectors.
Observed
BufferAllocationException: Maximum array size exceeded for this load factor (elements: 805306369, load factor: 0.750000) from IntIntHashMap.<init> in getNewOrdMapping; the same with assertions off.

All files of this repro: repro/LFV-458/

More files, environment, full output298 words

Put the file at lucene/core/src/test/org/apache/lucene/util/hnsw/TestHnswMergeLargeGraphOrdMapping.java. The vectors are FloatVectorValues.fromFloats(Collections.nCopies(N, new float[2]), 2); the stub graph reports size() = 805,306,369 and throws GraphCopyStarted from every other method, which a correct merger first calls when it starts copying the graph (maxConn()). The test uses no randomness, so the seed does not matter. It fails at once on Gradle's default 512 MB test heap (1.2 s on a 2 GB heap with a standalone runner).

As a regression test it stays the same test, but after a fix the mapping really allocates N-sized arrays and needs about 10 GB of heap (it passes with -Ptests.heapsize=14g), so upstream would likely mark it @Monster, like TestManyKnnDocs.

Not covered by the repro (code reading only): writing an 805M-vector segment, a real IndexWriter merge of it, and the tragic writer close.

Environment

lucenemain 645d4fe14a20; the merger-level runs also fail on 9.10.0, 9.12.3, 10.0.0, 10.2.0, 10.5.2 and 10.6.0; 9.9.2 completes the mapping
jdkOpenJDK 25.0.3
osLinux aarch64
Command (environment)
# from a Lucene checkout ($LUCENE), with the test in lucene/core/src/test/org/apache/lucene/util/hnsw/
./gradlew :lucene:core:test --tests org.apache.lucene.util.hnsw.TestHnswMergeLargeGraphOrdMapping -Ptests.seed=DEADBEEF
# assertions off fails the same way: add -Ptests.asserts=false
# with a fix applied the mapping allocates N-sized arrays: add -Ptests.heapsize=14g

Observed output

TestHnswMergeLargeGraphOrdMapping > testMergeReusesGraphOfMoreThan805MillionVectors FAILED
    junit.framework.AssertionFailedError: Unexpected exception type, expected GraphCopyStarted but got org.apache.lucene.internal.hppc.BufferAllocationException: Maximum array size exceeded for this load factor (elements: 805306369, load factor: 0.750000)
        at org.apache.lucene.tests.util.LuceneTestCaseParent.expectThrows(LuceneTestCaseParent.java:1437)
        at org.apache.lucene.util.hnsw.TestHnswMergeLargeGraphOrdMapping.testMergeReusesGraphOfMoreThan805MillionVectors(TestHnswMergeLargeGraphOrdMapping.java:75)

        Caused by:
        org.apache.lucene.internal.hppc.BufferAllocationException: Maximum array size exceeded for this load factor (elements: 805306369, load factor: 0.750000)
            at org.apache.lucene.internal.hppc.HashContainers.minBufferSize(HashContainers.java:95)
            at org.apache.lucene.internal.hppc.IntIntHashMap.ensureCapacity(IntIntHashMap.java:450)
            at org.apache.lucene.internal.hppc.IntIntHashMap.<init>(IntIntHashMap.java:107)
            at org.apache.lucene.internal.hppc.IntIntHashMap.<init>(IntIntHashMap.java:93)
            at org.apache.lucene.util.hnsw.IncrementalHnswGraphMerger.getNewOrdMapping(IncrementalHnswGraphMerger.java:221)
            at org.apache.lucene.util.hnsw.IncrementalHnswGraphMerger.createBuilder(IncrementalHnswGraphMerger.java:166)
            at org.apache.lucene.util.hnsw.IncrementalHnswGraphMerger.merge(IncrementalHnswGraphMerger.java:255)
BUILD FAILED

# -Ptests.asserts=false: the same failure
# trimmed: the app// frame prefixes, the seed frame, the inner expectThrows frame and the frames below merge:255 are left out

Why it looks unintended

  • No limit is documented or enforced below IndexWriter.MAX_DOCS, yet, by code reading, Lucene writes the segment it later cannot merge.
  • The cap came from #12950 (9.10.0), a speed-up that replaced a HashMap; 9.9.2 completes the same mapping.
  • #11905 fixed a smaller single-segment HNSW limit as a bug, and Test2BDocs force-merges MAX_DOCS plain documents (no vectors) into one segment.
The full argument177 words
  • No limit. IndexWriter.MAX_DOCS (IndexWriter.java:201-208, "Hard limit on the maximum number of documents that may be added to the index") is the only count limit. Lucene99HnswVectorsFormat validates only maxConn <= 512 and beamWidth <= 3200 (:102, :112, :271-283). HnswGraphMerger.merge is "Merge and produce the on heap graph" (HnswGraphMerger.java:46).
  • Written, then unmergeable (code reading). Lucene checks its intended limits before data is written (MAX_DOCS when a document is added); this one only appears one merge later.
  • Accidental origin. #12950 (49d521145de, backport 7d62b23ee90, 2023-12-15) replaced CollectionUtil.newHashMap(initGraphSize) with new IntIntHashMap(initGraphSize). The PR has no description; its only discussion is the author's own note on the hppc API. #14331 (3752dba9c51 on main, backport aa1e9f34b37 in 10.2.0) and #16794 moved the presize and kept it; neither discusses capacity.
  • Precedent. #11905 fixed an int overflow above about 16 million vectors in one segment and added the @Monster test TestManyKnnDocs.testLargeSegment (2,088,992 vectors, forceMerge(1)). Test2BDocs force-merges MAX_DOCS plain documents (no vectors) into one segment.
  • No test under lucene/*/src/test asserts this behaviour.
The devil's-advocate check243 words

The strongest case: one HNSW graph of 805M+ vectors is outside the supported envelope. In #15420 a committer (benwtrent) called merging tiny segments into one giant graph "antithetical to Lucene's design" and said "a single 6TB Lucene Segment is likely to hit other scaling issues first"; MAX_DOCS is index-wide, not a promise about one graph; IncrementalHnswGraphMerger and HnswGraphMerger are @lucene.experimental (ConcurrentHnswMerger has no tag of its own) and IntIntHashMap is @lucene.internal; the repro calls the merger with a stub graph.

Why it fails: #15420 is design advice in an RFC about another feature, not a documented limit or a won't-fix, and this failure is not limited to giant-plus-tiny merges. By code reading Lucene itself writes the segment (no count guard below MAX_DOCS; not run end to end), and Test2BDocs force-merges MAX_DOCS plain documents into one segment (the vector monster test reaches only 2,088,992 vectors, about 385x below this bound). The cap is an accident of an internal-structure swap that users cannot see or avoid except by keeping segments small. @lucene.experimental allows API changes, not merges that always throw, and by code reading the default IndexWriter path calls the same method with the same argument. The stub-graph limit stands: no IndexWriter run at this size exists. The argument bears on severity (low priority), not on intent. The cross-model fact check rated this case "moderate" and called the framing (crash, contrived, low priority) fair, provided "crash" means a merge exception, not corruption.

Suggested fix

Recommended Sorted (new doc, old ordinal) pairs and a merge walk instead of the hash map

Replace the hash map with one long per reused vector, (newDoc << 32) | oldOrd, sorted only if needed, and walk it alongside the merged vectors, both in new-doc order. The repro then passes with assertions on and off, and 25 HNSW, codec, kNN-query and reorder suites (577 tests) pass. It also uses less memory than the map.

All options and trade-offs4 options · 888 words

Option A was built and validated; B was checked with the repro only; C and D do not fix it.

Sorted (new doc, old ordinal) pairs and a merge walk instead of the hash map (validated by the checks) (recommended)

In IncrementalHnswGraphMerger.getNewOrdMapping, collect each reused graph's live vectors as (newDocId << 32) | oldOrd in a long[], sort it only if the pairs did not arrive in order, and walk it together with the merged vectors' iterator, one cursor per graph (+25/-6 lines). On main and branch_11x this is the only site; a branch_10_6 backport also needs ConcurrentHnswMerger.java:147.

diff --git a/lucene/core/src/java/org/apache/lucene/util/hnsw/IncrementalHnswGraphMerger.java b/lucene/core/src/java/org/apache/lucene/util/hnsw/IncrementalHnswGraphMerger.java
index 57aaaa54ac3..ccf45b8a92c 100644
--- a/lucene/core/src/java/org/apache/lucene/util/hnsw/IncrementalHnswGraphMerger.java
+++ b/lucene/core/src/java/org/apache/lucene/util/hnsw/IncrementalHnswGraphMerger.java
@@ -28,7 +28,6 @@ import org.apache.lucene.codecs.hnsw.HnswGraphProvider;
 import org.apache.lucene.index.FieldInfo;
 import org.apache.lucene.index.KnnVectorValues;
 import org.apache.lucene.index.MergeState;
-import org.apache.lucene.internal.hppc.IntIntHashMap;
 import org.apache.lucene.search.DocIdSetIterator;
 import org.apache.lucene.util.BitSet;
 import org.apache.lucene.util.Bits;
@@ -204,7 +203,9 @@ public class IncrementalHnswGraphMerger implements HnswGraphMerger {
   protected final int[][] getNewOrdMapping(
       KnnVectorValues mergedVectorValues, BitSet initializedNodes) throws IOException {
     final int numGraphs = graphReaders.size();
-    IntIntHashMap[] newDocIdToOldOrdinals = new IntIntHashMap[numGraphs];
+    // per graph: (new doc id << 32 | old ordinal) of each live vector, in new doc id order
+    final long[][] newDocIdAndOldOrdinal = new long[numGraphs][];
+    final int[] numMapped = new int[numGraphs];
     final int[][] oldToNewOrdinalMap = new int[numGraphs][];
     for (int i = 0; i < numGraphs; i++) {
       KnnVectorValues.DocIndexIterator vectorsIter = null;
@@ -218,26 +219,44 @@ public class IncrementalHnswGraphMerger implements HnswGraphMerger {
             vectorsIter =
                 graphReaders.get(i).reader.getFloatVectorValues(fieldInfo.name).iterator();
       }
-      newDocIdToOldOrdinals[i] = new IntIntHashMap(graphReaders.get(i).graphSize);
+      long[] pairs = new long[graphReaders.get(i).graphSize];
+      int count = 0;
+      boolean sorted = true;
       MergeState.DocMap docMap = graphReaders.get(i).initDocMap();
       for (int docId = vectorsIter.nextDoc();
           docId != NO_MORE_DOCS;
           docId = vectorsIter.nextDoc()) {
         int newDocId = docMap.get(docId);
-        newDocIdToOldOrdinals[i].put(newDocId, vectorsIter.index());
+        if (newDocId == -1) {
+          continue; // deleted
+        }
+        long pair = ((long) newDocId << 32) | vectorsIter.index();
+        sorted &= count == 0 || pair > pairs[count - 1];
+        pairs[count++] = pair;
+      }
+      if (sorted == false) {
+        Arrays.sort(pairs, 0, count);
       }
+      newDocIdAndOldOrdinal[i] = pairs;
+      numMapped[i] = count;
       oldToNewOrdinalMap[i] = new int[graphReaders.get(i).graphSize];
       Arrays.fill(oldToNewOrdinalMap[i], -1);
     }
 
+    // both sides are in new doc id order: walk them together, one cursor per graph
+    final int[] next = new int[numGraphs];
     KnnVectorValues.DocIndexIterator mergedVectorIterator = mergedVectorValues.iterator();
     for (int docId = mergedVectorIterator.nextDoc();
         docId < NO_MORE_DOCS;
         docId = mergedVectorIterator.nextDoc()) {
       int newOrd = mergedVectorIterator.index();
       for (int i = 0; i < numGraphs; i++) {
-        int oldOrd = newDocIdToOldOrdinals[i].getOrDefault(docId, -1);
-        if (oldOrd != -1) {
+        long[] pairs = newDocIdAndOldOrdinal[i];
+        while (next[i] < numMapped[i] && (int) (pairs[next[i]] >>> 32) < docId) {
+          next[i]++;
+        }
+        if (next[i] < numMapped[i] && (int) (pairs[next[i]] >>> 32) == docId) {
+          int oldOrd = (int) pairs[next[i]++];
           oldToNewOrdinalMap[i][oldOrd] = newOrd;
           if (initializedNodes != null) {
             initializedNodes.set(newOrd);

Trade-offs

  • 8 bytes per reused vector, temporary, against the hash map's 10.7-21.3 bytes (8 MB against 16.8 MB at 1M vectors); no capacity limit below MAX_DOCS, which is defined to stay below the JVM array limit (IndexWriter.java:205-208).
  • No hashing on the hot path; the sort (O(n log n)) runs only for a non-monotone doc map. Not benchmarked.
  • At 805M vectors the mapping still needs about 6.4 GB of pairs plus the existing 3.2 GB old-to-new array. The repro stops at the graph copy, so later limits in a real 805M-vector merge were not tested.

What the checks validated

  • The repro passes with -ea and with -Ptests.asserts=false (-Ptests.heapsize=14g, about 15 s each): all 805,306,369 ordinals are mapped and the merge stops at the graph copy. The vectors are virtual and the graph is a stub, so this shows the mapping, not a full 805M-vector merge.
  • 25 suites, 577 tests, 0 failures, 18 skipped: all of org.apache.lucene.util.hnsw and org.apache.lucene.codecs.lucene99, TestLucene104HnswScalarQuantizedVectorsFormat, TestLucene104ScalarQuantizedVectorsFormat, TestKnnGraph, TestHnswMergeAbort, TestSortingCodecReader, TestKnnFloatVectorQuery, TestKnnByteVectorQuery (lucene/core), TestBPReorderingMergePolicy and TestBpVectorReorderer (lucene/misc).
  • :lucene:core:check -x test passes (google-java-format, ecj lint, forbidden APIs).
  • The sort branch: no suite above reaches it (all pass with a throw there). Index-sorted merges map each segment's docs in its own order (MultiSorter.java:111-134); reordered merges can produce any permutation, but IndexWriter wraps those readers in a SortingCodecReader (IndexWriter.java:3560-3565), which is not an HnswGraphProvider, so their graphs are not reused (IncrementalHnswGraphMerger.java:113-115). A scratch probe with a random permutation doc map (1,000 vectors) gives the expected mapping on stock and on the fix. Open PR #16879 would bring non-monotone doc maps to this method.

A dense new-doc to old-ordinal int array (the finder's sketch)

A per-graph int[] indexed by new doc id, grown with ArrayUtil.grow. The repro passes with it on a 12 GB heap with -ea and -da; it was not run against the module suites.

Trade-offs

Each array grows to the graph's highest new doc id, so a small graph late in the merge order costs 4 bytes per document of the whole merged segment: a memory regression for ordinary merges (code reading).

Keep IntIntHashMap with load factor 0.99

new IntIntHashMap(n, 0.99).

Trade-offs

Only raises the ceiling to 1,063,004,405 entries (computed), still below MAX_DOCS, and slows probing. Not a fix.

Drop the presize

new IntIntHashMap() and let it grow.

Trade-offs

Does not work: growth stops at the same 2^30-slot table, so at most 805,306,369 keys (including the separately stored key 0) fit.

Downstream note

Engines built on Lucene that let operators force-merge very large vector indexes into one segment inherit this; it fails loudly at merge time and never changes search results or stored data.

How it was found

Found by code reading: while modelling HNSW merges, the finder checked every int-sized structure on the merge path against IndexWriter.MAX_DOCS and computed the exact bound with a small script that mirrors the hash map's sizing. A JUnit on the real merger confirmed it, and independent verifiers reproduced it with their own tests.

The method in full91 words

A hunt unit on HNSW parallel repair (a TLA+ model of the merge task protocol plus controlled-schedule runs) had, as a standing step, to check every int-sized structure on the merge path against IndexWriter.MAX_DOCS. The presized IntIntHashMap stood out; a short script mirroring HashContainers.minBufferSize gave the largest size that fits (805,306,368) and the smallest that throws (805,306,369), and a JUnit on the stock main jar with a stub graph of that size failed with -ea and -da (2026-10-26). The signal came from code reading, not from a formal tool's output.

Prior reports, history and checks

Prior reports

No prior report: GitHub issues, PRs and commits, downstream repos, four mailing lists, the JIRA archive and the web (118 tool queries plus manual searches, all 408 open PRs), searched 2026-10-26 and re-checked 2026-10-31. Closest: #15420, design advice against giant segments, not a report.

Searches5 searches

Verdict new. Related, not this bug: #12950 (introduced the map), #14331 and #16794 (moved the presize and kept it), #15420 (a committer's design stance against giant segments), #11905 (an earlier single-segment HNSW size limit, fixed as a bug), #16739 (same code, base selection), #16729 (same method, executor saturation), #16815 / #15904 (off-heap merge graphs, would make the bound easier to reach). No open PR changes the map (#16879, #16915, #16815, #15208 read; the draft #15208 renames the reader list but keeps the presize). Gaps: two downstream class-name queries hit a rate limit; mailing-list phrase matching is loose; private reports are not visible.

What was searched, when, and what turned up
WhereWhenHowFound
GitHub issues, PRs and commits (apache/lucene)2026-10-2645 issue/PR and 3 commit queries through the hunt tool, plus manual searches: the exception name and message, IntIntHashMap with capacity / overflow / merge / load factor, getNewOrdMapping and the map's field names, MAX_HASH_ARRAY_LENGTH, minBufferSize, the bound, "hnsw merge billion vectors"; all 22 items mentioning IntIntHashMapno report
Open PRs and recent filings2026-10-26the changed files of all 408 open PRs; the titles of 494 issues and PRs updated since 2026-09-01no open PR changes the map
Mailing lists (dev@, java-user@, issues@, commits@)2026-10-2648 queries, 12 per listonly commit mails carrying the hppc source
JIRA archive, web, downstream (Elasticsearch, OpenSearch and its k-NN plugin, Solr, hppc)2026-10-268 JIRA, 12 web and 2 downstream tool queries; manual exception-name and message searches; 4 code searchesno report; other bugs only
GitHub re-check (apache/lucene)2026-10-31"BufferAllocationException", "Maximum array size exceeded", "getNewOrdMapping", "IntIntHashMap merge"; open PRs touching the mergerno report ("getNewOrdMapping" returns #16729, #14331, #16739 and #16794, all related, none a report); #16879, #16915, #16555, #16815, #15208 do not change the map

History

Introduced in 9.10.0 by #12950 (49d521145de, 2023-12-15), which swapped a HashMap for the presized IntIntHashMap. Executed: 9.10.0, 9.12.3, 10.0.0, 10.2.0, 10.5.2, 10.6.0 and main fail; 9.9.2 completes the mapping. Not fixed upstream.

Versions tested8 rows

Introduced by #12950 "Use hppc IntIntHashMap to avoid Integer box/unbox when remapping vector ordinals during merge" (msokolov; 49d521145de on main, backport 7d62b23ee90, 2023-12-15; first release 9.10.0). Carried forward by #14331 (3752dba9c51 on main, backport aa1e9f34b37 first in 10.2.0: one presized map per reused graph and a private copy in ConcurrentHnswMerger) and #16794 (c50bc9523db, 2026-10-26, on main and so on branch_11x: both mergers share one site). An older, higher ceiling: 9.9.0-9.9.2 presized CollectionUtil.newHashMap, which fails from 1,610,612,672 nodes, so #12950 halved an existing ceiling. branch_10_6 and 10.6.0 keep both sites (:210, ConcurrentHnswMerger.java:147); branch_11x has main's single site; branch_10x was decommissioned upstream on 2026-10-29 (#16911). Upstream main 2f424fd428da, branch_11x d0c0decc660 and branch_10_6 91eaab4fb29 (2026-10-30) have the same code.

Versions and branches executed, with results
Version or commitVerdictResult
main 645d4fe14a20, ea57360912ed affectedthe repro fails with -ea and -da (one shared site since #16794)
branch_11x (d0c0decc660) affectedmain's single presize site (:221) [read]
10.6.0 (4ae946e5a9b, as branch_10_6 RC1) affectedfails with -ea and -da in both mergers; branch_10_6 keeps the same two sites [read]
10.5.1, 10.5.2 affected10.5.2 executed: fails in both mergers; default codec reuses graphs again (#16403)
10.4.0, 10.5.0 custom codec onlysame line [read]; with the default codec merges never reuse graphs (#16400, fixed by #16403 in 10.5.1), so only a non-per-field vectors format reaches it
9.10.0, 9.12.3, 10.0.0, 10.2.0 affectedfail with -ea and -da; the line is at all 19 tags 9.10.0-10.5.2
9.9.2 not affectedcompletes the mapping at 805,306,369 nodes (100 GB heap); 9.10.0 fails on the same input in 0.28 s
9.9.0-9.9.2 at 1,610,612,672 nodes older ceilingIllegalArgumentException from the presized HashMap's capacity overflow (executed on 9.9.2)

Independent checks

A cross-model review (OpenAI GPT-5.6), an independent reproducer with its own test and an executed boundary (805,306,368 maps, 805,306,369 fails), a refute-lens adversary, an intended-behaviour devil's advocate and an adjudication: confirmed, high confidence. Re-verified on the current main and 10.6.0.

Checks and verdicts221 words

This doc's fix validation (Gradle, scratch worktree) is under Suggested fix.

Each independent check and its verdict
CheckVerdictWhat it found
Earlier cross-model review real (high)OpenAI GPT-5.6, read-only, refute lens: traced the full merge path and found no earlier structure that rejects this count; its precision corrections were applied.
Reproducer realOwn public-API JUnit fails on main, 10.5.2, 9.12.3 and 9.10.0; executed boundary: 805,306,368 maps, 805,306,369 fails.
Cross-model review, refute lens not real (low), proceduralOpenAI GPT-5.6, read-only, could not run Java; agreed with the root cause and found no documented limit. A separate run executed an allocation-free bound test, which passes 4 of 4 and confirms the cap.
Intended-behaviour devil's advocate realBuilt and rebutted the "do not build segments this big" case.
adjudication confirmed, highRe-ran the independent reproduction's test with a fresh seed, -ea and -da.
re-verification still reproducesmain 645d4fe14a20 and branch_10_6 RC1 (= 10.6.0), -ea and -da.
noise audit one minimal testCut the evidence to one deterministic test of the default merger.
fact checks of this doc real; wording fixedAn independent fact check re-ran the repro, the Gradle commands and the fix and recomputed every bound; a cross-model fact check (OpenAI GPT-5.6) reviewed the text. Both confirmed the bug; their wording corrections are applied.

Diagram file

The diagram as a still image, generated with AI and checked against the code. It is under 10 MB, the limit for a GitHub attachment.

Sources and provenance

Finding record, Lucene and upstream sources, repro files and diagram notes1,463 words

Finding record

LFV-458, snapshot of 2026-11-06

Registry title
HNSW merge with graph reuse throws BufferAllocationException when the reused graph has more than 805,306,368 vectors
Area
search / HNSW (merge-time graph reuse: old-to-new ordinal mapping in IncrementalHnswGraphMerger and ConcurrentHnswMerger)
Priority
Low, rank 967: A default-codec HNSW merge that reuses a graph of more than 805,306,368 vectors throws on every attempt and, by code reading, closes the writer, so such a segment can never be merged; it is loud, but it needs only about 0.375 x MAX_DOCS vectors in one segment.
Impact class
crash
Impact note
Anyone on the default codec and default merger (9.10.0 to main; on 10.4.0 and 10.5.0 only with a codec whose vectors format is not wrapped per field, because there default-codec merges never reuse graphs) whose HNSW vector field has one segment of more than 805,306,368 vectors (about 0.375 x IndexWriter.MAX_DOCS, far past TieredMergePolicy's default 5 GiB merged-segment cap, so in practice built by forceMerge, a raised maxMergedSegmentMB, a custom merge policy or addIndexes, on a heap of tens to hundreds of GB) and who merges it again with its graph reused: that merge throws BufferAllocationException every time (executed at the merger API), and by code reading IndexWriter then treats it as a tragic event and closes the writer, losing uncommitted changes.
Summary
You merge an HNSW vector field in which one segment's graph has more than 805,306,368 vectors (Lucene writes such a segment without error, for example by merging smaller graphs) and expect the merge to reuse that graph as it does at any smaller size, because no HNSW vector-count limit is documented or enforced below IndexWriter.MAX_DOCS. Instead both mergers presize an IntIntHashMap to the graph size to map old to new ordinals, and that map's fixed 0.75 load factor and 2^30-slot table cap make its constructor throw BufferAllocationException ('Maximum array size exceeded for this load factor') before any other work. The merge fails again on every retry, so such a segment can never be merged with graph reuse; by code reading the IndexWriter is also closed as a tragic event.
Introduced
49d521145dede343c83614b2517380bf1bfc1d38 (main) and its branch_9x backport 7d62b23ee907f012b13399d9b056d0bf7c5c32bb, PR #12950 'Use hppc IntIntHashMap to avoid Integer box/unbox when remapping vector ordinals during merge' (msokolov; approved by benwtrent and gsmiller), which replaced CollectionUtil.newHashMap(initGraphSize) with new IntIntHashMap(initGraphSize) in IncrementalHnswGraphMerger.getNewOrdMapping, a performance-only change whose PR never discusses capacity; carried into both mergers by #14331 (3752dba9c51, branch_10x backport aa1e9f34b37, 10.2.0), which presizes one map per reused graph and gave ConcurrentHnswMerger its own copy of the line [read: git blame and git blame -w -C -C on main, git log -S, git tag --contains; exec: 9.9.2 completes the mapping at N = 805,306,369 with a 100 GB heap and 9.10.0 throws on the same input and heap] (2023-12-15 (main merge; the branch_9x backport has the same date and title))
First affected version
9.10.0 [exec] (the first release containing the branch_9x backport; 9.9.2, tagged later from the 9.9 bugfix branch, does not contain it and completes the mapping at N = 805,306,369 [exec]); the main commit is first released in 10.0.0
Affected versions
9.10.0 through 10.5.2, 10.6.0 (branch_10_6 RC1 4ae946e5a9b [exec], below) and main ea57360912ed; on 10.4.0 and 10.5.0 only with a codec whose vectors format is not wrapped per field (with the default codec their merges never reuse graphs: #16400, fixed by #16403 in 10.5.1) [read]. [exec, merger API with a synthetic reused graph] 9.10.0, 9.12.3, 10.0.0, 10.2.0, 10.5.2 and main fail with -ea and -da (both mergers on 10.2.0 and later; in 9.x and 10.0 the concurrent merger inherits the same method); [read] the presizing line is present at all 19 release tags from 9.10.0 to 10.5.2 (follow-up tag scan). Not affected at this size: 9.9.2 [exec, 100 GB heap], 9.9.0-9.9.1 (same code) [read], 9.6.0-9.8.0 (unsized HashMap) [read], before 9.6.0 (no merge-time graph reuse) [read]. Older, higher ceiling of the same presizing pattern: 9.9.0-9.9.2 throw IllegalArgumentException from 1,610,612,672 nodes because CollectionUtil.newHashMap's capacity overflows [exec on 9.9.2], so #12950 halved an existing ceiling; sizes 805,306,369 to 1,610,612,671 are a true regression. Not fixed upstream: on upstream main 645d4fe14a2 (after #16794) both mergers reach one presized map at IncrementalHnswGraphMerger.java:221; branch_10x and branch_10_6 keep both sites (IncrementalHnswGraphMerger.java:210, ConcurrentHnswMerger.java:147) [exec 2026-10-29: main 645d4fe14a2 and branch_10_6 RC1 4ae946e5a9b, -ea and -da; branch_10x 657014b3c57 identical code, read]. No end-to-end IndexWriter run with a real 805M-vector segment was done on any version.
Status
documented (brief, checks and a per-finding doc exist)
Verification
Confirmed by AI-run checks: an independent repro, a cross-model review that could only read the code (its low-confidence “not real” is procedural), an intended-behaviour check, a prior-report search and a skeptical audit; not yet triaged by a person.
Found by
non-formal: code reading
Upstream and prior reports
none found (2026-10-26, follow-up prior-report record verdict new: the scripted prior-report search with 118 queries (45 GitHub issue/PR on apache/lucene, 3 commit, 2 downstream-repo, 48 mailing-list over dev@, java-user@, issues@ and commits@, 8 JIRA-archive, 12 web; 277 distinct hits, 0 high mechanical overlap); 19 manual GitHub issue/PR searches over apache/lucene and downstream repositories, 4 code searches and about 40 issue, PR, comment and diff reads; the changed files of all 408 open apache/lucene PRs and the titles of 494 recent issues; plus the finder's run (21 queries) and the intended-behaviour check's (20 mailing-list and about 12 GitHub queries); 20 hits classified, none the same bug); not fixed upstream: #16794 (merged 2026-10-26, after the tested commit) removed ConcurrentHnswMerger's private copy, so on upstream main 49483e82b357 both mergers reach the single presized map at IncrementalHnswGraphMerger.java:221 (code reading; executed on main 645d4fe14a2 on 2026-10-29: both mergers fail there), branch_10x and branch_10_6 keep both sites, and no open PR changes the map (#16815, #15208 and #16879 read); related, not reports of this bug: #12950 (introduced the map; capacity not discussed), #14331 (copied the presize into both mergers), #16739 and #16729 (same code, other problems), #15420 (a committer's design stance against merging tiny segments into one giant graph), #11905 (an earlier single-segment HNSW size limit fixed as a bug); gaps: two downstream class-name queries hit the shared rate limit (the exception-name and symptom queries in those repositories returned 0), mailing-list phrase matching is loose, and private reports to security@lucene.apache.org are not visible
Lucene commit tested
main 645d4fe14a20 (Gradle in a scratch worktree and the standalone JUnit runner, assertions on and off); merger-level runs on 9.9.2 (100 GB heap), 9.10.0, 9.12.3, 10.0.0, 10.2.0, 10.5.2, 10.6.0 (its release commit 4ae946e5a9b, as branch_10_6 RC1) and main ea57360912ed; the recommended fix with 25 suites (577 tests) and the core static checks. OpenJDK 25.0.3, Linux aarch64, 2026-10-26 to 2026-10-31.

Lucene and upstream sources

Finding record
  • lucene/core/src/java/org/apache/lucene/util/hnsw/IncrementalHnswGraphMerger.java:133-144 (addReader: base needs deletePct <= 40 and the largest live count; graphs without deletions are added to graphReaders), :169-171 (createBuilder calls getNewOrdMapping), :192-235 (getNewOrdMapping), :209 (newDocIdToOldOrdinals[i] = new IntIntHashMap(graphReaders.get(i).graphSize))
  • lucene/core/src/java/org/apache/lucene/util/hnsw/ConcurrentHnswMerger.java:90-92 (createBuilder calls getNewOrdMapping), :142 (IntIntHashMap newIdToOldOrdinal = new IntIntHashMap(initGraphSize))
  • lucene/core/src/java/org/apache/lucene/internal/hppc/HashContainers.java:33 (DEFAULT_LOAD_FACTOR = 0.75f), :48 (MAX_HASH_ARRAY_LENGTH = 0x80000000 >>> 1), :81-99 (minBufferSize throws BufferAllocationException above 2^30 slots); IntIntHashMap.java:92-107, :446-455 (the constructor calls ensureCapacity, which computes minBufferSize before allocating)
  • lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99HnswVectorsWriter.java:518-524 (addReader for every merge reader with the field, then merger.merge) and :667-688 (createGraphMerger: IncrementalHnswGraphMerger by default, ConcurrentHnswMerger with a merge executor or more than one merge worker); util/hnsw/HnswGraphMerger.java:29 (@lucene.experimental), :46 ('Merge and produce the on heap graph'); index/IndexWriter.java:202-208 (MAX_DOCS = Integer.MAX_VALUE - 128)
The finding in 30 seconds
  • lucene/core/src/java/org/apache/lucene/util/hnsw/IncrementalHnswGraphMerger.java:110-147, (at 645d4fe14a20) :164-166, :204-250, :221
  • lucene/core/src/java/org/apache/lucene/util/hnsw/ConcurrentHnswMerger.java:78-79 (at 645d4fe14a20)
  • lucene/core/src/java/org/apache/lucene/internal/hppc/HashContainers.java:33, (at 645d4fe14a20) :48, :56-63, :81-99
  • lucene/core/src/java/org/apache/lucene/internal/hppc/IntIntHashMap.java:85-107, (at 645d4fe14a20) :120-124, :139-140, :446-455
  • lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99HnswVectorsWriter.java:510-525, (at 645d4fe14a20) :668-688
  • lucene/core/src/java/org/apache/lucene/codecs/perfield/PerFieldKnnVectorsFormat.java:198, (at 645d4fe14a20) :357-369
  • lucene/core/src/java/org/apache/lucene/codecs/lucene104/Lucene104Codec.java:123 (at 645d4fe14a20)
  • lucene/core/src/java/org/apache/lucene/index/IndexWriter.java:2286-2301, (at 645d4fe14a20) :5043-5057, :5109-5113, :6098-6106
  • lucene/core/src/java/org/apache/lucene/index/SegmentMerger.java:123-167; (at 645d4fe14a20) lucene/core/src/java/org/apache/lucene/codecs/KnnVectorsWriter.java:133-158; lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99HnswVectorsWriter.java:433-475
Who is affected
  • lucene/core/src/java/org/apache/lucene/index/TieredMergePolicy.java:52-58, (at 645d4fe14a20) :77, :82, :384-385; lucene/core/src/java/org/apache/lucene/index/IndexWriter.java:3103, :3275
  • a Lucene clone releases/lucene/10.4.0 and 10.5.0 IncrementalHnswGraphMerger.java:90-95 (checks the wrapper); releases/lucene/10.5.1 :108-111; releases/lucene/9.10.0 :71-78 (graphs with deletions skipped at :78); releases/lucene/10.3.2 :78-81 (:81)
Why it looks unintended
  • lucene/core/src/java/org/apache/lucene/index/IndexWriter.java:201-208 (at 645d4fe14a20)
  • lucene/core/src/java/org/apache/lucene/codecs/lucene99/Lucene99HnswVectorsFormat.java:102, (at 645d4fe14a20) :112, :271-283
  • lucene/core/src/java/org/apache/lucene/util/hnsw/HnswGraphMerger.java:29, (at 645d4fe14a20) :46
  • lucene/core/src/test/org/apache/lucene/document/TestManyKnnDocs.java:39, (at 645d4fe14a20) :61-86; lucene/core/src/test/org/apache/lucene/index/Test2BDocs.java:64-72
Suggested fix
  • upstream PR #16879 diff (read-only gh pr diff, 2026-10-31: addReader composes source doc maps)
History
  • a Lucene clone: git rev-parse releases/lucene/10.6.0 = 4ae946e5a9b; git tag --contains 7d62b23ee90; releases/lucene/10.4.0, 10.5.0, 10.5.1 IncrementalHnswGraphMerger.java addReader
  • upstream main 2f424fd428da (read-only gh api commits for the merger and hppc files, 2026-10-31)
Diagrams
  • lucene/core/src/java/org/apache/lucene/util/hnsw/IncrementalHnswGraphMerger.java:221; (at 645d4fe14a20) lucene/core/src/java/org/apache/lucene/internal/hppc/HashContainers.java:33, (at 645d4fe14a20) :48, :81-99

Other statements trace to the hunt's working files (models, logs, reviews and briefs), which are not published.

Repro files

About the diagrams

A picture of the mechanism, made in parallel with this doc from the code at 645d4fe14a20 and the executed repro; the PNG is the summary still.

Built 2026-11-06 from the hunt's record of this finding and its doc file. Lucene main under test in this hunt: ea57360912ed. Every listed finding was re-run on Lucene main e1faeba08bde on 2026-11-02.