CheckIndex reports an index clean after its HNSW graph test fails, unless failFast is set

On main, CheckIndex at level 2 or 3 includes an HNSW graph sub-test (#13984) that rejects four kinds of broken graph: a node out of range, a neighbour that is not on the node's level, neighbours out of order, and repeated neighbours. When it finds one and failFast is off (the default, and the only mode of the command-line tool), it prints test: hnsw graphs.........ERROR: ... and a stack trace, stores the exception in hnswGraphsStatus.error and returns. testSegment turns a stored sub-test error into a failed segment through its "Rethrow the first exception we encountered" chain, which has a clause for every other sub-test but none for the HNSW one, and nothing else reads hnswGraphsStatus.error. So the segment is never counted as bad: the run ends with "No problems were detected with this index.", Status.clean true, numBadSegments 0, exit code 0, and -exorcise does nothing. With failFast on, the same index throws CheckIndexException, so the setting meant to choose between stopping early and checking more segments decides whether the damage counts at all. No release has the sub-test; main and branch_11x do, so the bug would first ship in 11.0.0. Search, reads, merges and writes are unaffected; the wrong output is CheckIndex's verdict.

1. Level gate. :1114: if (level >= Level.MIN_LEVEL_FOR_INTEGRITY_CHECKS) (= 2, :4754); inside it :1143-1144: segInfoStat.hnswGraphsStatus = testHnswGraphs(reader, infoStream, failFast);.
2. The sub-test stores its failure. testHnswGraphs (:2926-2966) calls testHnswGraph for every HnswGraphProvider field, which throws CheckIndexException for the four conditions (:3016-3024 node range, :3031-3042 neighbour not on its level, :3046-3057 out of order, :3058-3065 repeated). Its catch (:2956-2964): if failFast, rethrow; else print ERROR: + the exception, set status.error, print the stack trace, return.
3. The verdict never sees it. :1158-1197, under "Rethrow the first exception we encountered / This will cause stats for failed segments to be incremented properly", rethrows the stored error of live docs, field infos, norms, postings, stored fields, term vectors, doc values, points, vectors (:1185-1187), index sort and soft deletes. There is no hnswGraphsStatus clause, and nothing in lucene/ reads hnswGraphsStatus.error.
4. No error, no bad segment. Only the outer catch (Throwable t) (:1200-1211) sets segInfoStat.error, prints FAILED and "exorciseIndex() would remove reference to this segment"; it is never reached. processSegmentInfoStatusResult (:975-985) counts a segment as bad only when segmentInfoStatus.error != null, otherwise keeps it for -exorcise's new commit.
5. Clean verdict. :933-934 sets clean = true when no segment is bad; :957-958 prints "No problems were detected with this index."; doCheck exorcises only when clean == false and returns clean ? 0 : 1 (:4902-4927). In the repro the HNSW error is the only problem; another bad segment, or an invalid segment-name counter (:945-955), still makes clean false.
6. Who runs with failFast off. The field defaults to false (:515); the command-line tool has no failFast option (:4787-4819); TestUtil.checkIndex(dir) and checkIndex(dir, level) pass false (TestUtil.java:302-308), and BaseDirectoryWrapper.close uses them (:44-45). Only MockDirectoryWrapper.close checks with failFast=true (:917); the stack traces quoted in #14127 and #15467 both run through it.
7. Nuance (code reading, not run). The sub-test catches Exception only, so any Exception raised during the walk is stored and masked: the four CheckIndexExceptions, and also, for example, an IOException from the graph reader. An Error still reaches the outer catch (Throwable t) and fails the segment. That splits the node-range check: testHnswGraph first puts every node of the level into a FixedBitSet (:3006-3010), whose set asserts the index is in range (FixedBitSet.java:296). With assertions on, an out-of-range node throws an AssertionError before the range check at :3016 and fails the segment (not masked); with assertions off it reaches that check, or an ArrayIndexOutOfBoundsException, and is masked. A neighbour id outside the graph meets the same assertion in FixedBitSet.get (:286). The three conditions that were run (repeated, out-of-order and off-level neighbours, all with in-range ids) are masked with assertions on and off. Only fields whose vectors reader is an HnswGraphProvider are checked (:2941-2943).

1,000 vectors of dimension 8 written by the stock default codec, one neighbour edited in .vex at the same byte length, CRC32 footer recomputed:

The study's finder showed the same for "neighbors out of order for node 0: 85<170", through a codec-SPI KnnVectorsFormat whose graph view lists neighbours out of order. With several threads a failFast check gets the graph error wrapped, as CheckIndexException: Segment _0 check failed. (:919-924; the diagram probe, 4 threads).

• Default vs opt-in. The level is opt-in: the programmatic default is 0 (the level field has no initialiser, CheckIndex.java:499) and the command-line default is 1 (:4748); the usage text says "Use -level with value of '2' or higher if you also want to check segment file contents". failFast off is the default. Two downstream engines' shard checks (Elasticsearch and OpenSearch Store.checkIndex) do not call setLevel, so the HNSW sub-test does not run there.
• What happens. A broken HNSW graph passes a deep health check: clean verdict, exit code 0, nothing for -exorcise to remove. The ERROR line and the stack trace are printed, so a person reading the whole log can see it; a script or caller that reads Status.clean or the exit code cannot.
• When a graph fails the check. Only after something else went wrong: a writer bug (two reached main in 2025, #14127 and #15467, both fixed in the writer; the stack traces in both issues run through the test framework's failFast check in MockDirectoryWrapper.close), a checksum-consistent edit of index files, or a valid 9.12.0-10.1.0 graph that repeats a neighbour (the companion bug LFV-543, where the masked "clean" happens to be right). Main's stock writer sorts each neighbour list and drops repeats before writing it (Lucene99HnswVectorsWriter.java:575, :585-587); that says nothing about other codecs or external writers.
• Scale. None needed: 4 vectors reproduce it, and every segment whose graph check fails with an Exception is masked the same way.
• What it does not do. No hit, score, document or file changes; with failFast off the graph error does not escape checkIndex(), it is only printed and stored.
• How likely: rare. It takes a deep check without failFast on main and a graph that is already broken.

TestUtil.getDefaultKnnVectorsFormat() is Lucene99HnswVectorsFormat(DEFAULT_MAX_CONN, DEFAULT_BEAM_WIDTH, 0) (TestUtil.java:1426-1428), so even a 4-document segment gets a graph. Node 0's level-0 list is the first entry after the .vex index header, stored as vInts 3, 1, 1, 1 (count, then deltas: neighbours 1, 2, 3); setting the second delta to 0 gives 1, 1, 2. CodecUtil.writeFooter writes a fresh checksum, so test: check integrity passes. The edit stands in for a writer bug; main's writer de-duplicates neighbours, so this is not the old-index case of the companion bug.

The assertNotNull on hnswGraphsStatus.error makes sure the HNSW sub-test really failed, so the test cannot fail for another reason. 20 of 20 random seeds fail the same way with assertions on.

Regression test: this same test; it passes with the fix and fits as one more method in TestCheckIndex.

cp $REPRO/TestCheckIndexHnswGraphError.java $LUCENE/lucene/core/src/test/org/apache/lucene/index/
cd $LUCENE && ./gradlew :lucene:core:test --tests TestCheckIndexHnswGraphError -Ptests.seed=DEADBEEF
# the same result with -Ptests.asserts=false

# gradle, stock 645d4fe14a20, -Ptests.seed=DEADBEEF (assertions on; the same with -Ptests.asserts=false)
TestCheckIndexHnswGraphError > testBrokenHnswGraphIsNotReportedClean FAILED
    java.lang.AssertionError: expected clean=false after the HNSW graph test failed, got clean=true:
    Checking index with threadCount: 1
    0.00% total deletions; 4 documents; 0 deletions
    ...
        test: vectors.............OK [1 fields, 4 vectors] [took 0.024 sec]
        test: hnsw graphs.........ERROR: org.apache.lucene.index.CheckIndex$CheckIndexException: Field "v" has repeated neighbors of node 0 with value 1
    org.apache.lucene.index.CheckIndex$CheckIndexException: Field "v" has repeated neighbors of node 0 with value 1
    	at org.apache.lucene.index.CheckIndex.testHnswGraph(CheckIndex.java:3059)
    	...
    No problems were detected with this index.

    Took 0.096 sec total.
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:3E199C6F7C78FE7D]:0)
        ...
        at org.apache.lucene.index.TestCheckIndexHnswGraphError.testBrokenHnswGraphIsNotReportedClean(TestCheckIndexHnswGraphError.java:75)

# with either fix: 1 test, 0 failures (assertions on and off)

• The contract. Status.clean: "True if no problems were found with the index." (CheckIndex.java:130). doCheck: "@return 0 iff the index is clean, 1 otherwise" (:4884). The tool: "This tool exits with exit code 1 if the index cannot be opened or has any corruption, else 0." (:4661-4662, usage text :4867-4868).
• failFast means "stop early". setFailFast: "If true, just throw the original exception immediately when corruption is detected, rather than continuing to iterate to other segments looking for more corruption." (:517-520).
• The chain says what it is for. "Rethrow the first exception we encountered / This will cause stats for failed segments to be incremented properly" (:1158-1159).
• The PR that added the check called these errors. In #13984 the author (Ben Chaplin) wrote "I chose not to throw an exception for disconnectedness" and then "So overall, exceptions are thrown if:" followed by the four conditions. A reviewer (Michael McCandless) called the missing coverage "a blind spot for CheckIndex today, which is scary". Nobody discussed the final verdict or the rethrow chain.
• The same omission was fixed before. Before LUCENE-9662 (#128, 424192e1704, 2021), testVectors and testSort stored their errors but the chain stopped at points; #128 added the Vectors, Index Sort and Soft Deletes clauses (a reviewer asked "were we failing to CheckIndex soft deletes before this?"). The HNSW sub-test arrived in 2025 with the same catch-and-store pattern and no clause.
• Tests. No upstream test references hnswGraphsStatus or testHnswGraphs, and none asserts a verdict after a failed HNSW check.

The strongest case: the HNSW block is a diagnostic ("Add HNSW graph checks and stats to CheckIndex" in CHANGES); the ERROR and stack trace are printed, so the problem is reported; level 2 is opt-in and CheckIndex is @lucene.experimental; and the masking is what keeps valid 9.12.0-10.1.0 indexes, whose repeated neighbours upstream calls "benign" (#14127), from being declared broken and exorcised.

Why it fails: the author separated the informational statistic (connectedness) from the four conditions that throw, and those throw the same CheckIndexException as every other integrity sub-test. The documented outputs (clean, the exit code, -exorcise) are what scripts and callers read, and the run contradicts itself ("ERROR", then "No problems"). failFast is documented only as stopping early, and the default mode is no misuse. The old-index point is an interaction with the companion bug, not intent: the remedy proposed upstream for old graphs is an index version gate (msokolov on #14127: "re-enable the check, with an index version gate"), not ignoring the sub-test, and the masking also hides real writer bugs. The study's reviewers that weighed intent, including cross-model reviews, found the intended-behaviour case weak. The cross-model fact check of this doc rated it moderate overall, because it differs by condition: for repeated neighbours the backward-compatibility argument has real weight (valid 9.12.0-10.1.0 graphs can hold them, and the rethrow alone made -exorcise drop 994 of 1,000 documents), which is why the recommended fix pairs the rethrow with a version gate; for out-of-order and off-level neighbours, and out-of-range nodes, no such argument exists. A fair upstream framing: count real structural failures as bad segments, and decide separately how to treat old graphs that repeat a neighbour.

The recommended option is the rethrow clause plus the companion bug's version gate; it and the rethrow alone were both run through gradle. The rethrow alone is not enough: it turns a valid old index into a broken one for CheckIndex, and -exorcise then deletes its documents. The warning variant was not run. Doc nit for the same PR: the HnswGraphsStatus.error javadoc says "Exception thrown during term index test" (CheckIndex.java:449).

Add the missing clause after the vectors clause of testSegment's rethrow chain, and in testHnswGraphs tolerate repeated neighbours in segments whose writer version is before 10.2.0 (graph builders before 10.2 could link a neighbour twice, #14127; de-duplicated at write time since #14157). The gate is the companion bug's fix: credit msokolov, who proposed "re-enable the check, with an index version gate" on #14127, and iverase's reduced test there. One file, CheckIndex.java.

// testSegment, after the vectors clause
} else if (segInfoStat.hnswGraphsStatus.error != null) {
  throw new CheckIndexException(
      "HNSW graph test failed", segInfoStat.hnswGraphsStatus.error);
}

// testHnswGraphs: pre-10.2 segments may repeat a neighbour
boolean allowRepeated =
    reader instanceof SegmentReader sr
        && sr.getSegmentInfo().info.getVersion().onOrAfter(Version.LUCENE_10_2_0) == false;
testHnswGraph(hnswGraph, fieldInfo.name, status, allowRepeated);
// testHnswGraph: } else if (nbr == lastNeighbor && allowRepeated) { continue; }

Trade-offs

• The gate keys on the writer version, so a repeated neighbour in a pre-10.2 segment caused by some other bug passes too; the other three conditions stay strict for every segment.
• Conservative, not exact. The known affected writers are 9.12.0, 9.12.1, 10.0.0 and 10.1.0 (9.11.1, 9.12.2, 9.12.3 and 10.2.2 wrote no repeated neighbour in 60 seeds each; registry LFV-543), but the gate tolerates repeats in every segment written before 10.2.0.
• Only for a SegmentReader. The public static testHnswGraphs(CodecReader, ...) applies the gate only when it gets a SegmentReader; CheckIndex itself always passes one.
• Maintainers may prefer a warning over a gate (option C); the split of the four conditions in #13984 is theirs to revisit.

What the checks validated

Applied to a scratch worktree of 645d4fe14a20 and run through gradle with OpenJDK 25.0.3, seed DEADBEEF (2026-10-31; the worktree was removed afterwards):

• stock: the minimal test fails with assertions on and off;
• with the fix: the minimal test passes with assertions on and off;
• the closest core suites, TestCheckIndex, TestAllFilesCheckIndexHeader, TestKnnGraph, org.apache.lucene.util.hnsw.* and org.apache.lucene.codecs.lucene99.*: 19 classes, 357 tests, 0 failures, 12 skipped;
• backward-codecs TestBasicBackwardsCompatibility (deep CheckIndex on the shipped 9.x and 10.x indexes): 816 tests, 0 failures, 2 skipped;
• :lucene:core:checkGoogleJavaFormat, ecjLintMain and forbiddenApisMain pass;
• the companion's old-index tests on the stock jars with CheckIndex shadowed by the fix: valid indexes written by 9.12.0, 10.0.0 and 10.1.0 whose graphs repeat a neighbour pass, and CheckIndex without failFast reports the 10.1.0 one (1,000 documents) clean, so -exorcise drops nothing (4 of 4 pass).

Not run: x86.

The first part of the recommended option: three lines after the vectors clause of the rethrow chain.

Trade-offs

• Unsafe alone. On a valid 10.1.0-written index (1,000 documents, a graph that repeats a neighbour) CheckIndex without failFast then reports clean=false badSegments=1 lostDocCount=994, and exorciseIndex leaves 6 of the 1,000 documents; on the stock jars the same test keeps all 1,000.
• Upstream tests would not notice: the shipped 9.12.0, 10.0.0 and 10.1.0 back-compat indexes hold no repeated neighbours, and TestBasicBackwardsCompatibility passes 814 of 814 with this option alone.

What the checks validated

The minimal test passes with assertions on and off (gradle, same scratch worktree; the first run without assertions died before any test when another process stopped the shared Gradle daemons, and was re-run).

Count the error for node range, neighbour level and order, and only print repeated neighbours, which upstream calls benign (#14127, #15467).

Trade-offs

No version gate is needed, but CheckIndex would no longer catch new duplicate-writing bugs like #14127 and #15467 on current segments. Not run.

Method: non-formal (the finder tagged it semi-formal). A main-side differential probe indexed vectors with release jars, then read, merged and checked the indexes on main. On a 10.1.0-written index whose graph repeats a neighbour (the companion bug), CheckIndex at level 2 printed test: hnsw graphs.........ERROR: ... has repeated neighbors followed by "No problems were detected with this index." with clean=true.

• Root cause by reading testSegment's rethrow chain and testHnswGraphs' catch block.
• Confirmation. A 2-test JUnit (an out-of-order graph through a codec-SPI vectors format, and the 10.1.0 index) failed on the stock main jars with assertions on and off; an independent reproducer, cross-model reviews from another model family (OpenAI GPT-5.6), a devil's-advocate verifier and an adjudication confirmed it, and a 2026-10-29 minimization cut the repro to the one test above.