The HNSW vectors formats accept 0 merge workers with a merge executor, then the first graph-building merge throws and closes the IndexWriter

Lucene99HnswVectorsFormat(maxConn, beamWidth, numMergeWorkers, mergeExec) lets an application merge HNSW graphs with several threads from its own ExecutorService. The constructor rejects maxConn and beamWidth outside their range and one worker together with an executor, but stores any numMergeWorkers <= 0. With a non-null executor the writer sends every graph-building merge to ConcurrentHnswMerger without looking at the count, and HnswConcurrentMergeBuilder sizes its worker array by it: for 0 the first read of workers[0] throws ArrayIndexOutOfBoundsException, for -1 the allocation throws NegativeArraySizeException. IndexWriter.merge treats that as a tragic event, so the writer closes and loses its uncommitted changes; every new writer with that setting fails the same way at its first such merge. Lucene104HnswScalarQuantizedVectorsFormat, HnswBitVectorsFormat and, since 10.6.0, the sandbox's DedupHnswVectorsFormat and DedupHnswScalarQuantizedVectorsFormat copy the same partial check. The same 0 without an executor merges serially on 10.2.0 and later. Opt-in and loud: the default codec (1 worker, no executor) is not affected, the last commit stays intact and CheckIndex is clean. Every release from 9.9.0 to 10.6.0 and main.

• Vectors format: the codec component that writes a field's kNN vectors and its HNSW graph. Lucene99HnswVectorsFormat is the default one; Lucene104HnswScalarQuantizedVectorsFormat (core), HnswBitVectorsFormat (lucene/codecs) and the sandbox's DedupHnswVectorsFormat and DedupHnswScalarQuantizedVectorsFormat (since 10.6.0) take the same merge arguments.
• numMergeWorkers, mergeExec: "number of workers (threads) that will be used when doing merge" and the ExecutorService they run on (Lucene99HnswVectorsFormat.java:215-219). The default constructor uses 1 worker and no executor (:120, :177-185).
• Tragic event: an unexpected exception inside IndexWriter that makes it roll back to the last commit and close itself; later calls throw AlreadyClosedException.

1. No lower bound. The constructor rejects maxConn and beamWidth outside their range ("must be positive ...", Lucene99HnswVectorsFormat.java:271-284) and numMergeWorkers == 1 && mergeExec != null (:289-292), then stores the count as is (:293). 0 and -1 pass. Lucene104HnswScalarQuantizedVectorsFormat.java:181, HnswBitVectorsFormat.java:125, DedupHnswVectorsFormat.java:185 and DedupHnswScalarQuantizedVectorsFormat.java:201 have the same check; both Dedup formats fail the same way (executed on main and 10.6.0).
2. The executor decides the merger, not the count. Lucene99HnswVectorsWriter.createGraphMerger returns a ConcurrentHnswMerger with the format's count whenever mergeExec != null (Lucene99HnswVectorsWriter.java:674-676). Only the other path, the merge scheduler's intra-merge executor, is guarded by > 1 (:678); otherwise the merge is serial (:688).
3. An array of 0 workers. HnswConcurrentMergeBuilder does workers = new ConcurrentMergeWorker[numWorker] (HnswConcurrentMergeBuilder.java:153). For -1 that throws NegativeArraySizeException; for 0 the array is empty.
4. Worker 0 is read. Joining the segments' graphs goes through runInBatches, whose first statement reads workers[0].batchSize (:374-375): ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0. Without graphs to join, finish() reads workers[0] (:478) and fails the same way; that is the site on 10.4.0-10.6.0.
5. The writer closes. IndexWriter.merge catches any Throwable and calls tragicEvent(t, "merge") (IndexWriter.java:5109-5113).

Two committed segments of one 2-dimensional vector each, tinySegmentsThreshold 0, SerialMergeScheduler, forceMerge(1), on main (the diagram probe and the fact check's probe): (0, executor) throws ArrayIndexOutOfBoundsException, (-1, executor) throws NegativeArraySizeException, and after either the writer is closed (isOpen() == false, the merge exception as its tragic exception); (0, null) and (2, executor) merge fine. With the default 4-argument constructor, two 1000-vector segments fail the same way (the independent reproduction, the diagram probe) and two 20-vector segments build no graph and merge.

• Who is exposed. Code that builds one of the five formats with an explicit numMergeWorkers and an ExecutorService, and computes or forwards the count: availableProcessors() / 2 on a one-CPU container gives 0, or a 0 / -1 "auto" default is passed through while an executor is always supplied. On 9.11.0-10.1.x even (0, null) fails, through the merge scheduler's intra-merge executor.
• Opt-in. The default codec builds new Lucene99HnswVectorsFormat(): 1 worker, no executor (Lucene104Codec.java:123). All five formats are @lucene.experimental.
• Effect. Loud and immediate: the first graph-building merge throws and the writer closes as a tragic event, losing its uncommitted changes. In the security probe forceMerge(1) threw 31-232 ms after it was called, with 600 committed documents kept and 50 uncommitted ones lost; with ConcurrentMergeScheduler the indexing thread saw the error within 0.6 s of starting to index, as "this writer hit an unrecoverable error" or AlreadyClosedException. Every new writer with that setting fails the same way at its first such merge. The minimal repro commits both documents first, so it shows the crash and the closed writer, not a loss.
• Not affected. The last commit stays intact and CheckIndex is clean; a valid setting merges the same index afterwards; other writers on the same executor are unaffected; the probe observed no hang, no resource growth and no wrong search results.
• Scale. None: two one-vector segments with threshold 0, or, with the default constructor (tinySegmentsThreshold 100), any merge large enough to build a graph: a merge of n vectors builds one when n > (int) (100 * ln n), from 648 vectors on (Lucene99HnswVectorsWriter.java:453, :712-722). Two 1000-vector segments fail; a 40-vector merge builds no graph and passes.
• Likelihood: rare. Nobody sets 0 workers on purpose, and no report was found in the documented searches.

One class, one method, in lucene/core (59 lines with the license header). TestUtil.alwaysKnnVectorsFormat makes the format the only vectors format; the public 5-argument constructor with tinySegmentsThreshold 0 makes two one-vector segments enough to build graphs (with the 4-argument constructor the same failure needs a merge large enough to build a graph under the default threshold, for example two 1000-vector segments); SerialMergeScheduler runs the merge on the test thread. It does not assert the closed writer: one symptom suffices.

Before it goes into a PR, two edits are needed or ./gradlew :lucene:core:check fails on the test: rename the unused catch parameter e to _ (ecj lint: "The value of the exception parameter e is not used"), and use new SameThreadExecutorService() instead of Executors.newSingleThreadExecutor(), which Lucene's forbidden-APIs check rejects ("Spawns threads with vague names"). The edited test still fails on stock main and passes with either fix, and the module checks then pass. The suggested regression test is this same test.

# main 645d4fe14a20, ./gradlew :lucene:core:test --tests ...TestHnswZeroMergeWorkers -Ptests.seed=DEADBEEF -Ptests.asserts=true
# (-Ptests.asserts=false: the same failure); middle stack frames elided
TestHnswZeroMergeWorkers > testZeroMergeWorkersWithExecutor FAILED
    java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:7EC450BD9E747D5B]:0)
        at org.apache.lucene.util.hnsw.HnswConcurrentMergeBuilder.runInBatches(HnswConcurrentMergeBuilder.java:374)
        at org.apache.lucene.util.hnsw.HnswConcurrentMergeBuilder.joinGraphs(HnswConcurrentMergeBuilder.java:286)
        at org.apache.lucene.util.hnsw.HnswConcurrentMergeBuilder.build(HnswConcurrentMergeBuilder.java:243)
        at org.apache.lucene.util.hnsw.IncrementalHnswGraphMerger.merge(IncrementalHnswGraphMerger.java:260)
        at org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsWriter.buildAndWriteGraph(Lucene99HnswVectorsWriter.java:525)
        ...
        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:5081)
        ...
        at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:2229)
        at org.apache.lucene.codecs.lucene99.TestHnswZeroMergeWorkers.testZeroMergeWorkersWithExecutor(TestHnswZeroMergeWorkers.java:55)
:lucene:core:test (FAILURE): 1 test, 1  failure

# 10.4.0, 10.5.2, 10.6.0 release jars: the same exception from HnswConcurrentMergeBuilder.finish (10.6.0: line 244)

• The contract. @param numMergeWorkers number of workers (threads) that will be used when doing merge (Lucene99HnswVectorsFormat.java:215-216, :232-233, :254-255; the same in the other four formats). No range, no @throws; no javadoc, CHANGES or MIGRATE entry gives 0 or a negative count a meaning.
• Validation is the constructor's style. maxConn and beamWidth "must be positive" (:271-284) and (1, executor) is rejected (:289-292); TestLucene99HnswVectorsFormat.testLimits asserts these (:64-74), and so does the quantized format's (TestLucene104HnswScalarQuantizedVectorsFormat.java:273-295); the bit format's testLimits checks only maxConn and beamWidth (TestHnswBitVectorsFormat.java:117-124).
• Thread counts below 1 are rejected elsewhere: "maxThreadCount should be at least 1" (ConcurrentMergeScheduler.java:145-146).
• Inconsistent. (0, null) merges serially on 10.2.0+ (the > 1 guard, Lucene99HnswVectorsWriter.java:678, from #14331); open PR #16871 also reads numMergeWorkers <= 1 as serial.
• Out of proportion. An accepted constructor argument fails later, at an unrelated merge, as an internal array exception that closes the writer.
• History. The worker array and the partial check came together in #12660 (9.9.0); no review of #12660, #13124/#13190 or #14331 discusses counts below 1. One upstream test passes 0 with a null executor as a placeholder (backward-codecs TestLucene99HnswQuantizedVectorsFormat.testLimits, five calls that never merge), so do not claim that no test passes 0.

Strongest case (refute review, the cross-model review, the intended-behaviour check, adjudication): 0 or -1 worker threads is nonsense input to an @lucene.experimental expert API; two committers said the count should not be a user setting at all (benwtrent in a #13124 PR discussion comment, 2024-02-23; mikemccand on luceneutil #507, 2026); and the 5-argument constructor the repro uses is labelled "(This is a Test-Only Constructor)".

Why it fails: the constructors are public, and the 4-argument one, not test-only, fails the same way on a 2000-vector merge; the constructor already validates its neighbouring arguments; the value works without an executor; and the failure closes the writer late instead of raising an argument error at configuration time. If maintainers prefer to remove the knob, that resolves the report too. Verdict: not intended, a small missing-validation defect; no checker before the cross-model fact check rated the intended-behaviour argument better than weak; that check rated it moderate.

No-noise check. The hunt does not claim behaviour that needs absurd parameters and then fails loudly. For noise: 0 or -1 worker threads is nonsensical, and the failure is loud. Why it is still claimed, as a low-priority missing-validation bug only: 0 is an accepted value of this constructor (with a null executor it merges serially on 10.2.0+), so adding an executor turns an accepted value into a crash; a 0 comes from ordinary computed settings such as availableProcessors() / 2 on a one-CPU container; and the failure is late and destructive (at the first graph-building merge, closing the writer and dropping uncommitted documents), not an argument error at configuration time where the neighbouring arguments are checked. The cross-model fact check (OpenAI GPT-5.6) judged it "claim-low, not noise", if filed as a narrow missing-validation hardening bug, not as corruption, security or broad availability impact.

All options were run through gradle on a scratch worktree of 645d4fe14a20 (seed DEADBEEF) unless marked. The adjudication advised A or C over B.

+5 lines in each of Lucene99HnswVectorsFormat, Lucene104HnswScalarQuantizedVectorsFormat, HnswBitVectorsFormat and the sandbox's DedupHnswVectorsFormat and DedupHnswScalarQuantizedVectorsFormat, just before the existing (1, executor) check; together they say "with an executor, use at least 2 workers". Keeps (0, null) and (-1, null), which merge serially today on 10.2.0+, and touches no upstream test. Applied to only the first three, it would leave the two Dedup formats broken.

if (numMergeWorkers < 1 && mergeExec != null) {
  throw new IllegalArgumentException(
      "numMergeWorkers must be at least 1 when a merge executor is passed; numMergeWorkers="
          + numMergeWorkers);
}
if (numMergeWorkers == 1 && mergeExec != null) { // existing check, unchanged

Trade-offs

• An application that passes (0, executor) today and never builds a graph in a merge would now fail when it builds its codec.
• The backward-codecs formats share the partial check, but their production fieldsWriter throws UnsupportedOperationException, so they need no change; writable subclasses of them exist only in backward-codecs tests.
• One combined check mergeExec != null && numMergeWorkers <= 1 would also work but changes the existing message. The stale javadoc (option D) fits the same change.

What the checks validated

• Three-format version: the repro passes with assertions on and off; 534 tests in 27 suites pass (the package wildcards org.apache.lucene.codecs.lucene99.*, org.apache.lucene.codecs.lucene104.*, org.apache.lucene.util.hnsw.*, TestKnnGraph; org.apache.lucene.codecs.bitvectors.*, which include 54 postings, segment-info and for-util tests), 18 skipped, 0 failures; :lucene:core:check :lucene:codecs:check -x test passes with the PR-ready test.
• Five-format version: applies cleanly; the repro passes with assertions on and off; 272 tests in 8 suites (the repro, the three format suites and the four org.apache.lucene.sandbox.codecs.dedup.* suites) pass, 12 skipped, 0 failures; :lucene:core:check :lucene:codecs:check :lucene:sandbox:check -x test passes; the Dedup probe passes with it.
• The whole core module was not run.

One line in Lucene99HnswVectorsWriter.createGraphMerger, which all five formats write through, so it covers every format that delegates to that writer: (0, executor) then means what (0, null) means.

if (mergeExec != null && numMergeWorkers > 1) {
  return new ConcurrentHnswMerger(

Trade-offs

It silently ignores an executor the caller passed, and (1, executor) stays rejected while (0, executor) is accepted. It can be combined with A as a second guard.

What the checks validated

• The repro passes with assertions on and off (the merge runs).
• 342 tests in 17 suites (lucene99.*, util.hnsw.*) and 553 tests in 24 suites (lucene104.*, TestHnswMergeAbort, TestKnnGraph, bitvectors.*, backward-codecs lucene99.* and lucene102.*) pass, 0 failures; google-java-format, ecj lint and forbidden APIs pass on core.
• With it shadowed in, the Dedup probe's two merges run.

The hunt's first fix sketch, in the two core formats.

Trade-offs

Breaks today's working (0, null) on 10.2.0+; copied into the backward-codecs constructors it would make five placeholder calls of TestLucene99HnswQuantizedVectorsFormat.testLimits pass for the wrong reason.

What the checks validated

The repro passes with it (standalone runner, the fix shadowed in, assertions on and off); no gradle suite was run.

"If larger than 1, a non-null ExecutorService must be passed as mergeExec" has been wrong since #13124/#13190 (9.11.0): a null executor falls back to the merge scheduler's intra-merge executor. mikemccand already pointed this out on luceneutil #507.

Trade-offs

Javadoc only; not run.

Method: non-formal (code reading; no model checker, SMT solver, prover or bounded-exhaustive harness produced the signal).

• A hunt unit (the upstream change #16794 to the concurrent HNSW merge), 2026-10-26, on main a9779ef3c193. While building a differential harness for that merge, the finder noticed that HnswConcurrentMergeBuilder sizes its worker array by numMergeWorkers and that the only check on the count in any layer is numMergeWorkers == 1 with an executor.
• It wrote a 4-test public-API JUnit (0 and -1 workers with an executor, the scalar-quantized format, the (0, null) control); 3 of 4 failed with assertions on and off, and a 4-line constructor check made them pass.
• Independent checkers then widened it: the tragic close of the writer, the default 4-argument constructor, (0, null) failing on 9.11.0-10.1.x, and every release from 9.9.0.