MemoryIndex.addField throws AssertionError for a KnnFloat16VectorField when assertions are enabled: a stale assertion in storeVectorValues predates the FLOAT16 branch

MemoryIndex.addField(IndexableField, Analyzer) stores a vector field through the private storeVectorValues (MemoryIndex.java:677-679, :793), whose first statement is assert vectorField instanceof KnnFloatVectorField || vectorField instanceof KnnByteVectorField; (:794). #13633 wrote it in 2024 (first released in 9.12.0), when those were the only vector field classes, and it was correct then. #16383 (2026-07-27, unreleased 11.0.0) added KnnFloat16VectorField, which extends Field and is not a KnnFloatVectorField, together with a case FLOAT16 arm in the same method (:819-840), the MemoryIndex reader's getFloat16VectorValues (:1782-1789) and MemoryFloat16VectorValues (:2425), but did not widen the assertion and added no MemoryIndex test. So with -ea, adding any float16 vector field to a MemoryIndex (directly, through fromDocument, or through Monitor.match on one document) throws a message-less AssertionError before the FLOAT16 arm runs. With -da the arm stores the vector and it reads back exactly. Only test runs are affected, and of the upstream refs inspected only main and branch_11x have the code. The fix is one line.

• storeVectorValues(Info, IndexableField) (MemoryIndex.java:793-865): the private method addField calls when the field type has vectorDimension() > 0 (:677-679). It switches on the field's VectorEncoding (:795), with one arm per encoding: BYTE (:796-817), FLOAT16 (:819-840) and FLOAT32 (:842-863). Each arm checks the field's class, allows one value per field, and copies the vector.
• KnnFloat16VectorField (lucene/core/.../document/KnnFloat16VectorField.java): public class KnnFloat16VectorField extends Field (:37, @lucene.experimental at :35), whose field type has VectorEncoding.FLOAT16 (:55).

1. addField (:620) creates the field's FieldInfo from the field type (getInfo at :622, createFieldInfo at :759-783), so the encoding is FLOAT16, and calls storeVectorValues (:678). Every fromDocument overload loops over the document's fields calling addField (:571-573).
2. The method's first statement is assert vectorField instanceof KnnFloatVectorField || vectorField instanceof KnnByteVectorField; (:794). For a KnnFloat16VectorField both are false, so with -ea it throws java.lang.AssertionError (no message) before any vector value is stored. The field is already registered by then: getInfo put its Info and FieldInfo into the field map (:740-744), so a caller that catches the error is left with a field that FieldExistsQuery matches and getFloat16VectorValues returns null for (executed probe).
3. With -da the assertion is skipped: the FLOAT16 arm checks instanceof KnnFloat16VectorField, copies the short[] into the field's float16 state (:1273, :1279), and the reader returns it through getFloat16VectorValues (:1782-1789) and MemoryFloat16VectorValues (:2425).
4. The assertion adds no check the arms lack for the vector field classes: each arm already throws IllegalArgumentException for a field of the wrong class (:813-816, :836-839, :859-862). Its only other effect, with -ea, is that a custom IndexableField with vector attributes that is none of the listed classes fails there with AssertionError instead of the arm's IllegalArgumentException (read).

One field, one dimension, value Float.floatToFloat16(1f): with -ea, 1 test run, 1 failure at MemoryIndex.java:794 (gradle and standalone, seeds DEADBEEF and 5EED1234); with -da, 1 run, 0 failures and an exact read-back. Nothing depends on data, size, codec or threads.

• Who is hit. Code that adds a KnnFloat16VectorField to a MemoryIndex on a JVM with -ea: MemoryIndex.addField, every MemoryIndex.fromDocument overload, and lucene/monitor's Monitor.match(Document, ...) for a single document, which builds a SingletonDocumentBatch that calls addField for every field (DocumentBatch.java:107-113). A public-API probe executed for this doc: with -ea, Monitor.match throws AssertionError from MemoryIndex.java:794 via DocumentBatch$SingletonDocumentBatch.<init>(DocumentBatch.java:110) and DocumentBatch.of(DocumentBatch.java:59); with -da the registered query matches (1 match). Multi-document Monitor batches go through IndexWriter and are not affected (read). The highlighters feed MemoryIndex token streams, not fields, so they are not affected (read).
• When. Test runs. Lucene's gradle build enables assertions for about 75% of project seeds by default (TestsAndRandomizationPlugin.java:240-248), and LuceneTestCase requires -ea to match -Dtests.asserts (TestRuleAssertionsRequired.java:36-54). Any application test suite run with -ea sees it too: a test that adds a valid float16 field through one of these paths fails.
• Production is fine. With -da the vector is stored and read back exactly at every affected commit tested (bfe5c847d3c, ea57360912ed, 645d4fe14a20). A service running production JVMs with -ea (unusual) would lose one operation per document with the field; AssertionError is an Error, so catch (Exception e) does not catch it (executed probe).
• Why Lucene's tests miss it. lucene/memory/src/test has no float16 test, and no test elsewhere puts a float16 field into a MemoryIndex (read, grep).
• Scale. Deterministic: every float16 vector field that can be constructed (any dimension and similarity; the constructor itself rejects a null or empty vector, a null similarity and a zero vector with COSINE, KnnFloat16VectorField.java:39-53), on the first call. Unreleased: main and branch_11x only, of the refs inspected; KnnFloat16VectorField is @lucene.experimental. No downstream product was tested; one downstream project is still discussing whether to expose float16 at all (elastic/elasticsearch#163176).

The test sits in org.apache.lucene.index.memory next to TestMemoryIndex; it uses only public API, so it needs no package-private access. The failure is Lucene's own message-less AssertionError, so the comment line on the failing call states expected versus actual. Pass -Ptests.asserts=true explicitly: without it the build picks assertions from the seed.

As the regression test: the same test, with its comment line wrapped (below), as the first float16 test in lucene/memory/src/test/org/apache/lucene/index/memory/; if the maintainers prefer, the same method can go into the existing TestMemoryIndex next to its float32 and byte vector tests (:769, :824). Its one comment line is longer than 100 columns, which google-java-format (:lucene:memory:check) flags; with the line wrapped the module's checks pass and the test behaves the same.

Not shown by this test, same root cause, executed elsewhere: MemoryIndex.fromDocument (the finder's second test) and Monitor.match on one document (a reach probe).

cp TestMemoryIndexFloat16Vector.java $LUCENE/lucene/memory/src/test/org/apache/lucene/index/memory/
cd $LUCENE && ./gradlew :lucene:memory:test --tests org.apache.lucene.index.memory.TestMemoryIndexFloat16Vector -Ptests.seed=DEADBEEF -Ptests.asserts=true
# with -Ptests.asserts=false the test passes: the bug needs assertions enabled

# gradle, main 645d4fe14a20, -Ptests.seed=DEADBEEF -Ptests.asserts=true
TestMemoryIndexFloat16Vector > testAddFloat16VectorField FAILED
    java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:C69D0D8C0A2877EA]:0)
        at org.apache.lucene.index.memory.MemoryIndex.storeVectorValues(MemoryIndex.java:794)
        at org.apache.lucene.index.memory.MemoryIndex.addField(MemoryIndex.java:678)
        at org.apache.lucene.index.memory.TestMemoryIndexFloat16Vector.testAddFloat16VectorField(TestMemoryIndexFloat16Vector.java:30)

ERROR: 1 test has failed:
  - org.apache.lucene.index.memory.TestMemoryIndexFloat16Vector.testAddFloat16VectorField (:lucene:memory)

# -Ptests.asserts=false: 1 test, 0 failures
# 10.5.2 and 10.6.0 release jars: does not compile (cannot find symbol: class KnnFloat16VectorField)

• Deliberate support. #16383's MemoryIndex.java diff (+110/-0) adds the KnnFloat16VectorField import, the case FLOAT16 arm with its own one-value and wrong-class checks, the float16 Info state, getFloat16VectorValues and MemoryFloat16VectorValues with a scorer. The unchanged assertion on the method's first line contradicts all of it.
• The migration note asks for this update. lucene/MIGRATE.md:629-638 ("IEEE FLOAT16 vector APIs (GITHUB#16383)"): "Custom readers and exhaustive VectorEncoding switches need a FLOAT16 arm." This switch got its arm; its guard did not.
• History. The assertion came with #13633 (9e831ee8097, 2024-08-07, "Add float|byte vector support to memory index", first released in 9.12.0), correct when only two vector field classes existed. #16383 did not touch it, and none of its review discussion mentions MemoryIndex. The earlier, unmerged fp16 PR #15549 had the same omission.
• No exclusion anywhere. The addField javadoc (MemoryIndex.java:613-619) names no unsupported field type; no CHANGES.txt or MIGRATE.md entry excludes MemoryIndex; no test expects the error, and lucene/memory/src/test has no float16 test. TestMemoryIndex tests the float32 and byte vector fields (:769, :824 and their round-trip helpers).
• Elsewhere it works. IndexWriter indexes the same float16 field with -ea (verifiers' controls).
• Precedent. #15334 fixed an NPE in the MemoryIndex vector path as a bug although MemoryIndex has no kNN search (lucene/CHANGES.txt:1493-1494, 10.4.0).

The case for intended: (1) the failure is assertion-only, so production with -da is unaffected; (2) KnnFloat16VectorField is @lucene.experimental and float16 support is openly incomplete (tracker #16725); (3) MemoryIndex runs no kNN search for any encoding (searchNearestVectors are no-ops, MemoryIndex.java:1800-1811; TestMemoryIndex.java:819: "we don't really do knn search right now for MemoryIndex"), so its vector support is partial anyway; (4) the assertion could be read as a precondition that only the float32 and byte classes may reach storeVectorValues.

Why it fails: (a) the float32 and byte classes carry the same @lucene.experimental tag (KnnFloatVectorField.java:38, KnnByteVectorField.java:38) and are supported and tested in MemoryIndex; (b) if float16 were meant to be excluded, #16383 would not have written a FLOAT16 arm, state, reader and values class for MemoryIndex; (c) the missing kNN search applies to every encoding and concerns searching, not storing; (d) nothing supports the precondition reading: the assertion predates the class it rejects; (e) "assertion-only" lowers the severity, not the validity: the assertion itself is the defect, and it fires in the configuration Lucene's tests use. The cross-model reviews rated the intended argument weak; the devil's advocate and the adjudication rejected it. A later cross-model fact check of this doc rated the same points moderate as a case that the defect is low-value and not worth an issue of its own, and weak as a case that it was designed so: two different questions. Verdict: not intended; a low-severity oversight.

Both options were run through gradle in a scratch worktree of 645d4fe14a20 (seed DEADBEEF). Either way, add the minimal test as the regression test.

MemoryIndex.java:794; KnnFloat16VectorField is already imported at :40.

   private void storeVectorValues(Info info, IndexableField vectorField) {
-    assert vectorField instanceof KnnFloatVectorField || vectorField instanceof KnnByteVectorField;
+    assert vectorField instanceof KnnFloatVectorField
+        || vectorField instanceof KnnFloat16VectorField
+        || vectorField instanceof KnnByteVectorField;
     switch (info.fieldInfo.getVectorEncoding()) {

Trade-offs

• The smallest change; keeps the original guard and its meaning ("one of the vector field classes").
• A future vector encoding could fall into the same trap.

What the checks validated

• The repro passes with assertions on and off (1/1 each); it fails on stock main with assertions on.
• All lucene/memory tests with assertions on: 4 classes, 52 tests, 0 failures (TestMemoryIndex 30, TestMemoryIndexAgainstDirectory 16, TestSlicedIntBlockPool 5, the repro 1).
• :lucene:memory:check -x test --continue with the fix and the comment-wrapped test passes (google-java-format, ecj lint and forbidden-apis on main and test sources).

Remove line :794. Dropping it loses no check for the vector field classes: each arm already throws IllegalArgumentException for a field of the wrong class (:813-816, :836-839, :859-862).

Trade-offs

• The next encoding cannot be blocked by it, and a custom vector field that is none of the three classes gets the same IllegalArgumentException with -ea as it already does with -da.
• Removes a guard some reviewers may want to keep.

What the checks validated

• The repro and TestMemoryIndex with assertions on: 2 classes, 31 tests, 0 failures.

Method: non-formal (a differential JUnit, not a model or a bounded-exhaustive check). The hunt unit on float16 vectors wrote edge probes that run a float16 field and its float32 twin through the same APIs and compare the outcomes. TestFp16Probes#testMemoryIndex failed with -ea and passed with -da; reading the failing frame (MemoryIndex.java:794) gave the root cause.

The finder wrote a two-test repro (addField, fromDocument): 2/2 fail with -ea, 2/2 pass with -da and with the one-line fix. It was then reviewed by a reviewer from another model family (OpenAI GPT-5.6, refute and works-as-designed lenses), reproduced by independent verifiers (one from a package outside Lucene, with float32 and byte controls), judged, traced through history, cut to the one-test repro on 645d4fe14a20, and reviewed again there.