MemoryIndex offers a StoredFieldVisitor fields that store nothing, and asks once per field name instead of once per value

MemoryIndex has returned stored fields since 9.5.0. Its StoredFields.document(int, StoredFieldVisitor) walks the index's fields by name and calls visitor.needsField once per name, before it checks whether that field has any stored value, then delivers all stored values of the name on that one YES. An index written by IndexWriter asks once per stored value and only for stored values. So on a MemoryIndex the visitor is offered fields that store nothing (indexed-only, points-only or doc-values-only), and a field stored N times gets one decision instead of N: LazyDocument throws ArrayIndexOutOfBoundsException for the unstored field or silently loses later values, and a visitor's per-value NO or STOP is ignored. document(int) and name-based visitors are unaffected; nothing is written or corrupted.

1. One entry per field name, whatever the field is. fields is a SortedMap<String, Info> (MemoryIndex.java:418). addField (:620) calls getInfo (:622), which creates an Info for every new name (:739-744), for indexed, points, doc-values and vector fields alike. Stored values are kept only when the field type is stored (:673-675); storeValues adds one entry per stored value (:867-885). So body (Store.NO) has an Info with storedValues == null, and tag has one Info with two values.
2. The read loop asks once per name, before looking at the values. for (Info info : fields.values()) (:2247) calls visitor.needsField(info.fieldInfo) (:2248), returns on STOP (:2249-2251), skips on NO (:2252-2254), and only then checks info.storedValues != null (:2255) and delivers every value of the name (:2256-2270).
3. What an index does. The default codec's reader loops over the document's stored values, for (int fieldIDX = 0; fieldIDX < doc.numStoredFields; fieldIDX++) (Lucene90CompressingStoredFieldsReader.java:694), and calls needsField for each one (:706). The hunt's exhaustive check found the same behaviour in all 15 index-backed reader configurations it ran other than ParallelLeafReader (default codec, SimpleText, wrappers, composite readers, merge paths). ParallelLeafReader, run separately, also asks once per stored value; its only protocol deviation is an unrelated STOP leak.
4. Why LazyDocument breaks. Each getField(fieldInfo) adds one lazy value for that field (LazyDocument.java:68-88). On first read it loads the document by name and assigns f.realValue = realValues[i] per lazy value (:107-122). For body, realValues is empty, so index 0 throws ArrayIndexOutOfBoundsException (:119). For tag only one lazy value was made, so t2 is never handed out; with assertions on, the assertion at :113-114 fires first (it runs in the wrong direction, LFV-697).
5. Not claimed: MemoryIndex delivers values grouped by name in sorted order, an index in the order added. No javadoc promises an order, and the fixes keep it.

With the recommended fix every row agrees.

• Who is hit. Code that calls StoredFields.document(int, StoredFieldVisitor) on a MemoryIndex with a visitor that decides per value ("first value", "up to N values", STOP after a value, counting calls) or expects only stored fields, and users of LazyDocument (lucene/misc) over a MemoryIndex. Any document with an unstored field or a field stored more than once triggers it.
• What they get. An ArrayIndexOutOfBoundsException from a lazy field, silently missing values (later values of a field with LazyDocument, assertions off), or values a visitor declined. Silent except for the ArrayIndexOutOfBoundsException and, with assertions on, the AssertionError LazyDocument raises instead of losing t2 (LFV-697). The registry's impact class is silent wrong results; for the per-value leg, which rests on reader behaviour rather than a written contract, the fairer framing is a reader-protocol inconsistency that can give per-value visitors silently wrong results.
• Who is not hit. document(int), document(int, Set<String>) / DocumentStoredFieldVisitor and other visitors that decide by name and accept every value. By code reading, the adjudication and the intended-behaviour check found that the highlighters' stored-field visitors and StoredFieldsWriter.MergeVisitor also decide by name, so by that reading their output is not affected (code reading only: the highlighter and monitor modules were not run, with or without the fix); UnifiedHighlighter's LimitedStoredFieldVisitor decides per value but enforces the same length budget in stringField, so only its skip shortcut is lost. In the Lucene tree, the highlighters and the monitor module use MemoryIndex for matching and offsets, not to read its stored fields back.
• Scale and likelihood. One in-memory document per MemoryIndex, so the effect is per read and per document. Reading MemoryIndex stored fields through a custom per-value visitor is niche, and the recorded searches found no report of it since 9.5.0: likelihood rare, priority low.

The test uses only public classes and methods (MemoryIndex.fromDocument, createSearcher, StoredFields.document(int, StoredFieldVisitor)) and sits in MemoryIndex's test package only to match the upstream test location. It uses no randomness: assertions on and off and every seed tried (DEADBEEF, 5EED7451, 1A2B3C4D) give the same result. The expected list [tag, tag] is what an IndexWriter index returns for the same document (each of the finder's two tests asserts that control first; on 645d4fe14a20 the control assertion passes before the MemoryIndex assertion of the same test fails). One assertion covers both legs: body is offered although it stores nothing, and tag is asked once for two values. The suggested regression test is this same test.

> Task :lucene:memory:test FAILED

TestMemoryIndexStoredFieldVisitorCalls > testNeedsFieldOncePerStoredValue FAILED
    java.lang.AssertionError: expected:<[tag, tag]> but was:<[body, tag]>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:BC27CAEB93919950]:0)
        at junit@4.13.2/org.junit.Assert.fail(Assert.java:89)
        at junit@4.13.2/org.junit.Assert.failNotEquals(Assert.java:835)
        at junit@4.13.2/org.junit.Assert.assertEquals(Assert.java:120)
        at junit@4.13.2/org.junit.Assert.assertEquals(Assert.java:146)
        at org.apache.lucene.index.memory.TestMemoryIndexStoredFieldVisitorCalls.testNeedsFieldOncePerStoredValue(TestMemoryIndexStoredFieldVisitorCalls.java:54)

• Reader contract. StoredFields.java:73-77: "Expert: visits the fields of a stored document, for custom processing/loading of each field."
• Visitor contract. StoredFieldVisitor.java:80-84: needsField is the "Hook before processing a field. Before a field is processed, this method is invoked so that subclasses can return a Status representing whether they need that particular field or not, or to stop processing entirely." Status.NO: "don't visit this field, but continue processing fields for this document" (:90). Each added value is its own field instance (tag is two StoredField objects), and the index-backed readers that were checked treat it that way (the default codec, SimpleText and the 15 configurations of the exhaustive check other than ParallelLeafReader, which also asks per value and differs only by an unrelated STOP leak).
• LazyDocument. LazyDocument.java:61-63: getField "must be called once for each value of the field name specified in sequence that the values exist". TestLazyDocument calls it from needsField (TestLazyDocument.java:203-215). On SOLR-4589 (2013), where that sentence comes from, a committer (hossman) wrote that getField "assumes it will be called exactly once for each StorableField that exists for that field name, in order".
• Design intent. LUCENE-3309 (#4382, 2011), which introduced the visitor: the reader "invokes this API once per field in the document being visited", and the name StoredFieldVisitor was asked for because "FieldVisitor sounds really general and like it visits all fields, not just stored ones".
• MemoryIndex's role. "a replacement/substitute for RAM-resident Directory implementations" (MemoryIndex.java:106). Earlier divergences of MemoryIndex from an index reader were fixed as bugs (#7462 / LUCENE-6402, #12109).
• History. The loop came with #11999 and is unchanged since. Its review discussed whether MemoryIndex should have stored fields and how to keep the values, never the call count or unstored fields; TestMemoryIndex reads stored fields only through document(0) (TestMemoryIndex.java:749). No inspected test, javadoc, release note or #11999 review comment says the per-name behaviour is intended.

Strongest case (the refute review, rated moderate): StoredFieldVisitor is expert and @lucene.experimental (StoredFieldVisitor.java:35), and no javadoc states how often needsField is called or that fields without stored values are left out. MemoryIndex keeps one entry per name, so one decision per name is a defensible design, and LazyDocument's once-per-value rule is a precondition on its caller.

Why it fails: it does not explain offering a field that stores nothing, which is hard to reconcile with "visits the fields of a stored document" (though no javadoc excludes such fields in so many words), and that leg alone makes LazyDocument throw. For the per-value leg, every index-backed reader that was checked calls per value, the visitor was designed to be invoked once per field in the document (LUCENE-3309), LazyDocument's own test relies on it, and MemoryIndex presents itself as a Directory substitute. The call count is not written down; hence the optional javadoc sentence with fix A, and the smaller fix B. The cross-model review and the intended-behaviour check judged the argument insufficient; the adjudication confirmed.

Three options. A was built and run through gradle in a scratch worktree of 645d4fe14a20 and through the standalone probe; B was built and run through the standalone runner; C is listed for completeness.

Move the needsField call inside the loop over info.storedValues, and skip an Info whose storedValues is null (MemoryIndex.java, one method, +23/-21, mostly re-indentation). Value order is unchanged, so A aligns the calls per value, not the whole visit trace (values still come grouped by name in sorted order). The skip test is storedValues == null, which holds exactly when no stored value was added (storeValues, MemoryIndex.java:867-885); checked with ordinary Lucene field classes only. Optional companion, not written or built: one javadoc sentence on StoredFieldVisitor#needsField saying it is called once for each stored value, only for fields with stored values.

for (Info info : fields.values()) {
  if (info.storedValues == null) {
    continue; // nothing stored for this field: do not offer it to the visitor
  }
  for (Object value : info.storedValues) {
    // one decision per stored value, like the index-backed readers
    StoredFieldVisitor.Status status = visitor.needsField(info.fieldInfo);
    if (status == StoredFieldVisitor.Status.STOP) {
      return;
    }
    if (status == StoredFieldVisitor.Status.NO) {
      continue;
    }
    // unchanged: binaryField / doubleField / floatField / longField / intField / stringField
  }
}

Trade-offs

• Matches what the checked index-backed readers do; document(int) and name-based visitors return the same document.
• Cost: one needsField call per stored value instead of per name, on a one-document index. No benchmark was run.
• Changes the per-value leg, whose call count no javadoc states; a maintainer may prefer B plus a javadoc decision. Compatibility: a visitor written against MemoryIndex's current behaviour that keeps state and relies on one decision per name would now be asked once per value; B avoids that change.

What the checks validated

• gradle, stock + test: fails (expected:<[tag, tag]> but was:<[body, tag]>); + fix: passes.
• gradle, + fix: the whole lucene/memory module (TestMemoryIndex 30, TestMemoryIndexAgainstDirectory 16, TestSlicedIntBlockPool 5, the repro 1: 52 tests, 0 failures), with assertions on and with assertions off.
• gradle, + fix and the test: :lucene:memory:check -x test passes (google-java-format, ecjLint, forbiddenApis).
• standalone probe with the fix: all 8 index-versus-MemoryIndex comparisons agree (LazyDocument, first-value and STOP visitors, call counts, document(0)).
• again, in a second, fresh scratch worktree: stock + test fails with assertions off; + fix: the whole lucene/memory module, the repro included, passes with assertions off and on (52 tests, 0 failures); the standalone runner with the fix passes the repro with -ea and -da on another seed.
• Not run: the full Lucene suite, nightly tests, the highlighter and monitor modules.

Add if (info.storedValues == null) continue; before the needsField call (+3 lines). Keeps one decision per name.

Trade-offs

Fixes the leg on the explicit contract and LazyDocument's exception with the smallest change. The multi-valued leg stays: LazyDocument still loses t2, first-value and STOP visitors still get both values. The repro then fails with expected:<[tag, tag]> but was:<[tag]>, so the regression test would need tag stored once.

What the checks validated

Standalone runner, main 645d4fe14a20, assertions off: the repro fails with [tag]; in the probe the unstored-field rows agree and the multi-valued rows still differ.

Say in the MemoryIndex javadoc that needsField is asked once per field name, for every field.

Trade-offs

Hard to reconcile with "visits the fields of a stored document", and leaves LazyDocument's exception in place.

Method: semi-formal (bounded-exhaustive translation validation), a hunt unit (StoredFieldVisitor protocol across implementations), 2026-10-27, main a9779ef3c193. The harness built single documents of 1-3 stored values over the names a, b, c and 6 stored types (6,174 documents), wrote each once with IndexWriter and once with MemoryIndex, and read each with every prefix-distinct YES/NO/STOP script, comparing the visit traces with the protocol all index-backed readers follow.

• 15 index-backed configurations (default codec, SimpleText, wrappers, composite readers, merge paths): 0 deviations. ParallelLeafReader, run separately, deviates only by an unrelated STOP leak.
• MemoryIndex: 80,892 of 89,802 visits differ; 7,020 of them differ only in the order of values across field names, which is not claimed.
• The alternative model "one needsField per field name, names in sorted order, all values of the name on YES" matches all 89,802 visits, which pins the mechanism.
• A follow-up probe found that an indexed, not stored field is offered too; the finding was then reduced to a public-API JUnit.