MemoryIndex returns stored StringField, KeywordField and numeric field values as bytes, because it stores binaryValue() instead of storedValue()

MemoryIndex, Lucene's one-document in-memory index, has supported stored fields since 9.5.0 (#11999). It does not ask a field what to store (IndexableField#storedValue(), which IndexWriter uses since 9.6.0, #12116); it keeps binaryValue() if that is non-null, else numericValue(), else stringValue(). For StringField and KeywordField built from a String, binaryValue() is the indexed term, and for IntField, LongField, FloatField and DoubleField it is the sortable point encoding. So these stored values come back as bytes, and Document.get or numericValue() returns null; a StoredField(String, StoredFieldDataInput) is dropped. IndexWriter returns the String, the number and the bytes for the same document. No error is raised and query matching is unchanged. Every release from 9.6.0 to 10.5.2 and main is affected; the fix is a small change to MemoryIndex.storeValues that uses storedValue().

• MemoryIndex (lucene/memory): a one-document index in memory. You add fields with addField(IndexableField, Analyzer) or build it with MemoryIndex.fromDocument(doc, analyzer), then search it through createSearcher(). Lucene Monitor uses one for every one-document batch.
• Stored fields: the original values kept with Field.Store.YES, read back with storedFields().document(docId). #11999 added them to MemoryIndex in 9.5.0.
• binaryValue() vs storedValue(): an IndexableField exposes its value through several getters. binaryValue() also serves indexing: for StringField/KeywordField it is the term to index, for IntField and the other numeric fields the encoded point. storedValue() (9.6.0, #12116) is the one value meant for stored fields: the String, the number or the bytes.

1. addField stores a field whose type says stored() by calling storeValues (MemoryIndex.java:673-675 on main a9779ef3c193).
2. storeValues (MemoryIndex.java:867-885) keeps field.binaryValue() if it is non-null, else numericValue(), else stringValue(). It never calls storedValue().
3. StringField(String, Store) sets binaryValue = new BytesRef(value) and a String storedValue (StringField.java:59-66); KeywordField(String, Store) the same (KeywordField.java:96-103). IntField.binaryValue() returns NumericUtils.intToSortableBytes of the value (IntField.java:84-88); LongField, FloatField and DoubleField likewise. So storeValues keeps bytes.
4. A StoredField(String, StoredFieldDataInput) returns null from all three old getters (Field.java:264-270, :419-434); only storedValue() exposes it (Field.java:607-624). So nothing is kept.
5. The stored-fields reader replays each kept object by its runtime type (MemoryIndex.java:2242-2273): a BytesRef becomes a binary field, so Document.get and numericValue() return null.
6. IndexWriter instead stores field.storedValue() and throws "Cannot store a null value" when it is null (IndexingChain.java:1491-1505).

Not affected (both agree): StringField(BytesRef), KeywordField(BytesRef), a StoredField built from a String, a number or bytes (tested: StoredField(String), StoredField(int), StoredField(BytesRef)), TextField(Store.YES). One expert constructor differs the other way round: for StoredField(String, CharSequence, FieldType) with a CharSequence that is not a String (a StringBuilder), IndexWriter throws, since Field.storedValue() throws IllegalStateException (Field.java:607-624), while MemoryIndex stores its toString() (executed in the fact check). For the String and numeric cases the type is wrong, but the bytes are the field's own UTF-8 term or point encoding and decode back to the value (executed for the example values "doc-1" and 5); only the StoredFieldDataInput value is lost.

• Who. Applications that read stored fields back from a MemoryIndex searcher: percolator- or Monitor-style matching that displays or highlights the matched document, or a custom Monitor CandidateMatcher that reads the document. #11999 added stored fields for exactly that use. A one-document Monitor batch is a MemoryIndex (DocumentBatch.java:44-46, :55-63, :103-114).
• Default vs opt-in. Stock field classes with Store.YES, the documented way to keep a value; no special setting. Only the StoredFieldDataInput case needs an @lucene.experimental API (10.2.0 on).
• Every time, silently. The outcome depends only on the field class and constructor: each affected class gave the same wrong form for all 100 random and edge values tried. No exception, no warning. (This probe and the next two ran on main ea57360912ed; lucene/memory and lucene/monitor are unchanged from there to a9779ef3c193.)
• Inconsistent results. In Monitor, a matcher that read a stored StringField("tenant", "t1", Store.YES) got null in a one-document batch and "t1" in a two-document batch (built with IndexWriter), for the same matching queries.
• It can spread. Copying the MemoryIndex reader into a real index with IndexWriter.addIndexes(SlowCodecReaderWrapper.wrap(...)) succeeds, CheckIndex reports it clean, and the copy keeps the wrong types.
• How bad. Wrong type, not lost data, for the String and numeric cases tested (the bytes are the field's own term or point encoding and decode back to the value); the StoredFieldDataInput value is lost. Matching is unchanged.
• How common. No in-tree module reads stored fields from a MemoryIndex (the highlighter fills one from token streams only), and no report was found in the recorded searches; one public question (2024) adds LongField(..., Store.YES) to a MemoryIndex but never reads it back.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<doc-1> but was:<null>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:120)
       org.junit.Assert.assertEquals(Assert.java:146)
       org.apache.lucene.index.memory.TestMemoryIndexStoredStringField.testStoredStringFieldRoundTrips(TestMemoryIndexStoredStringField.java:32)

• The contract. CHANGES 9.5.0: "GITHUB#11999: MemoryIndex now supports stored fields." (CHANGES.txt:3768). IndexableField#storedValue(): "Stored value. This method is called to populate stored fields and must return a non-null value if the field stored." (IndexableField.java:74-78). Field.Store.YES: "The value is stored in its original form, i.e. no analyzer is used before it is stored." (Field.java:596-599). CHANGES 9.6.0: "GITHUB#12116: Introduce IndexableField#storedValue() to expose the value that should be stored to IndexingChain without needing to guess the field's type." (CHANGES.txt:3569-3571).
• History: an oversight. The #12116 commit message (96136b42821) describes this exact failure: binaryValue() "is used both for sorted(_set) doc values and storage. This makes it impossible to implement keyword fields that is stored and doc-valued, as the field returns a non-null value for both binaryValue() and stringValue() and stored fields no longer know which field to store". It adds storedValue() so that a stored fields format cannot "mistakenly use binaryValue()/stringValue()/numericValue() instead of storedValue()". In lucene/memory it changed only TestMemoryIndex.java. No PR thread of #12116, #12054, #12139 or #14213 mentions MemoryIndex. #12139 did change MemoryIndex.java (cce33b07e4f, +30 lines): addField now indexes a field without a TokenStream by taking binaryValue() as its single term (a new storeTerm), with a new TestMemoryIndex.testKeywordWithoutTokenStream. It did not change storeValues: the indexing path was adapted to the new meaning of binaryValue(), the storing path was not.
• A regression for StringField. StringField(String, Store.YES) round-trips on 9.5.0 and fails on 9.6.0. On the 9.6.0 jars, StringField.java from just before the #12139 backport passes and from #12139 ("Faster indexing of string fields") fails.
• Tests. TestMemoryIndex.testStoredFields (TestMemoryIndex.java:719-766) asserts that numbers come back as numbers, strings as strings and bytes as bytes, but only with StoredField, whose binaryValue() is null for non-binary values. No test asserts the binary form for StringField, KeywordField or IntField.
• Precedent. #12109 ("MemoryIndex should not fail integer fields that enable doc values") fixed another MemoryIndex/IndexingChain divergence for the new IntField as a bug: it "aligns MemoryIndex with IndexingChain".

The strongest works-as-designed case, built by an intended-behaviour verifier and by a review from another model family: the MemoryIndex class javadoc says a query selects "on the field names and associated (indexed) tokenized terms, not on the original fulltext(s) - the latter are not stored but rather thrown away immediately after tokenization" (MemoryIndex.java:133-135); MemoryIndex never promises equivalence with IndexWriter; and StoredFieldDataInput is @lucene.experimental (StoredFieldDataInput.java:27).

Why it fails: that sentence is from 2005 (the original contribution 8f9e2a15e78 says "free text(s)"; 0062898ada6 reworded it later that year) and predates stored-field support; #11999 then made stored fields a documented feature with a type-preserving test. The behaviour changed in 9.6.0 through an unrelated indexing speed-up, and nothing documents a reduced stored-field contract. For StoredFieldDataInput, neither its javadoc nor #14213 limits it to IndexWriter, and MemoryIndex drops it silently instead of storing it or throwing. Verdict: not intended, a defect. Expect the class-javadoc sentence to be quoted back and answer with the #11999 CHANGES entry and testStoredFields; a reviewer may also read #12139's MemoryIndex edit as a sign that only indexing was meant to follow the new binaryValue(), though nothing documents that stored values were meant to change. Expect pushback mainly on priority (stored fields in MemoryIndex were accepted reluctantly in #11999) and on compatibility for custom fields (see the fix trade-offs).

Two variants of the same fix were applied in a scratch worktree at a9779ef3c193 and run through gradle (seed DEADBEEF); they differ only in what happens to a stored field whose storedValue() is null.

In storeValues, take field.storedValue() and add its value by StoredValue.Type; a DATA_INPUT value is read into a new BytesRef of length() bytes by a small readBytes helper (an IOException becomes an UncheckedIOException, since addField declares none). If storedValue() is null (IndexWriter rejects such a field), fall through to today's binaryValue() / numericValue() / stringValue() guess. +30/-0 in MemoryIndex.java.

StoredValue storedValue = field.storedValue();
if (storedValue != null) {
  switch (storedValue.getType()) {
    case INTEGER -> info.storedValues.add(storedValue.getIntValue());
    case LONG -> info.storedValues.add(storedValue.getLongValue());
    case FLOAT -> info.storedValues.add(storedValue.getFloatValue());
    case DOUBLE -> info.storedValues.add(storedValue.getDoubleValue());
    case STRING -> info.storedValues.add(storedValue.getStringValue());
    case BINARY -> info.storedValues.add(storedValue.getBinaryValue());
    case DATA_INPUT -> info.storedValues.add(readBytes(storedValue.getDataInputValue()));
  }
  return;
}
// ... the old binaryValue() / numericValue() / stringValue() guess, unchanged

Trade-offs

• The smaller of the two changes, because it keeps today's guess when storedValue() returns null: a custom stored IndexableField with a null storedValue() keeps working in MemoryIndex as today, although IndexWriter rejects it.
• A and B both change a stored field whose storedValue() throws: StoredField(name, aStringBuilder, StoredField.TYPE) now throws IllegalStateException in MemoryIndex, as it already does in IndexWriter; today MemoryIndex stores its toString() (executed in the fact check, stock and with A). A custom field whose storedValue() differs from its old getters changes too.
• Cost: the DATA_INPUT copy reads the input once, the same work IndexWriter's stored-fields writer does; BINARY values stay by reference, as today; the other types are boxed as before.
• Not tested with either fix: over-long stored strings, reuse of a BytesRef after addField returns, a StoredFieldDataInput that fails to read, custom fields with several non-null getters.

What the checks validated

• Stock main a9779ef3c193, gradle, seed DEADBEEF: the repro fails 7 of 8 with assertions on and with assertions off; the control passes.
• With A: the repro passes 8 of 8 with assertions on and with assertions off; all of lucene/memory passes (59 tests in 4 classes: TestMemoryIndex 30, TestMemoryIndexAgainstDirectory 16, the repro 8, TestSlicedIntBlockPool 5, per the fact check's JUnit XML) and all of lucene/monitor (181 tests, 1 skipped), and :lucene:memory:checkGoogleJavaFormat and :lucene:memory:ecjLint pass.
• The classpath runner with A shadowed on main: 8 of 8 pass. The hunt's earlier sketch of the same idea (it also copies BINARY values) passed the finder's repro, an independent 11-case suite (8 defect cases, 3 controls) and 51 upstream memory tests on ea57360912ed.

Same as the recommended option, but throw new IllegalArgumentException("Cannot store a null value") instead of the fallback; the old guess is removed. +28/-13 in MemoryIndex.java. It does not copy IndexWriter's other behaviour: there is no IndexWriter.MAX_STORED_STRING_LENGTH check (IndexingChain.java:1495-1502), and like A and today's code it keeps a BINARY value by reference, while IndexWriter's stored-fields writer copies the bytes when the document is added (Lucene90CompressingStoredFieldsWriter.java:303-319).

Trade-offs

One null rule for both writers and less code; #12116 made IndexWriter fail early on such a field ("If there is a problem ... it's currently discovered late"). But a custom stored field with a null storedValue() that works in MemoryIndex today would start to throw, so it is a visible behaviour change for such callers.

What the checks validated

• With B: all of lucene/memory passes (59 tests, the repro included) and all of lucene/monitor (181 tests, 1 skipped); format and lint pass. So no path that these suites ran with seed DEADBEEF adds a stored field with a null storedValue() to a MemoryIndex (B would throw).

Add StringField/KeywordField from a String, the four numeric fields and a StoredFieldDataInput to TestMemoryIndex.testStoredFields, or compare stored fields in TestMemoryIndexAgainstDirectory; this doc's repro can serve as the test.

Trade-offs

testStoredFields used only StoredField, whose binaryValue() is null for non-binary values, which is why the drift went unnoticed.

Index with Store.NO and add a StoredField under the same name, for example new StringField("id", "doc-1", Store.NO) plus new StoredField("id", "doc-1"), or new IntField("count", 5, Store.NO) plus new StoredField("count", 5). Checked on stock main: the term and exact queries match, get("id") = "doc-1" and numericValue() = 5.

Trade-offs

Needs a change in every application that builds such documents. For the StoredFieldDataInput case, pass the bytes as StoredField(String, BytesRef) instead; that constructor already agrees with IndexWriter.

Method: bounded-exhaustive translation validation on the stock jar (semi-formal tier), in a hunt unit that checked MemoryIndex against IndexWriter as the reference.

• The harness's stored-fields scope built each stored-value-producing field class (16 kinds x 1-2 values: 32 cases) into a single-field document, indexed it once with MemoryIndex and once in a one-document IndexWriter index, and compared what a StoredFieldVisitor saw.
• On the stock main jar it reported 14 STORED mismatches: StringField(String), KeywordField(String), IntField, LongField, FloatField, DoubleField and StoredField(StoredFieldDataInput), each with one and two values; the finder reports 0 with a storedValue()-based storeValues (no log of that run is kept; the diagram probe with the fix shadowed agrees with IndexWriter on all 13 cases).
• The finder traced the mismatches to MemoryIndex.storeValues, wrote a public-API JUnit with the IndexWriter value asserted first, and ran it on main and on releases back to 9.5.0. Independent verifiers then reproduced it with their own tests (see the checks).