MemoryIndex reports fields that index no terms (doc values, points, vectors, stored only) as indexed with positions and norms

MemoryIndex accepts fields that index no terms: doc values, points, vectors and stored-only values. For each such field name (when no instance of the name indexes terms), its reader builds a FieldInfo whose index options are hard-coded to DOCS_AND_FREQS_AND_POSITIONS (..._AND_OFFSETS with storeOffsets), so hasNorms() is true and getNormValues returns a norms iterator, although the field type says NONE, MemoryIndex stores no term, and IndexWriter records NONE with no norms for the same document. FieldInfo#getIndexOptions promises NONE "if the field is not indexed".

Search on the MemoryIndex is unaffected; the wrong metadata shows when the reader is copied into an index with IndexWriter.addIndexes(SlowCodecReaderWrapper.wrap(reader)). On 9.0.0 and later, next to a natively indexed copy of the document the writer's schema check throws, and into an empty index the copy succeeds but persists the wrong schema, after which ordinary addDocument calls with that field are rejected, in later writer sessions too, until the copied documents are deleted before they are merged with other documents and a new writer is opened, or the index is rebuilt. FieldExistsQuery on a stored-only or point-only field also matches on a MemoryIndex where a real segment throws.

The metadata has been wrong since 6.0.0, when MemoryIndex started accepting fields without terms; the exceptions appear from 9.0.0, when the writer's strict schema checks arrived. A fix in MemoryIndex (take NONE from the field type, switch to positions when a later instance of the name indexes terms, serve norms only when hasNorms(), and, as IndexWriter does, invert only fields whose type indexes something) passes the repro and the lucene/memory, lucene/monitor and lucene/highlighter test suites.

1. Accepting the field. addField(IndexableField, Analyzer) calls getInfo(field.name(), field.fieldType()) first (MemoryIndex.java:622), so every field gets a FieldInfo. A null token stream with indexOptions() == NONE stores no term (:636-645); only the doc value, point, stored value or vector is kept (:647-679).
2. One FieldInfo per name. getInfo builds the FieldInfo from the first instance of a name (:739-744); later instances only add point dimensions and the doc-values type (:745-755).
3. The hard-coded options (createFieldInfo, :759-783):

   IndexOptions indexOptions =
       storeOffsets
           ? IndexOptions.DOCS_AND_FREQS_AND_POSITIONS_AND_OFFSETS
           : IndexOptions.DOCS_AND_FREQS_AND_POSITIONS;

fieldType.indexOptions() is never read. The other field-type settings come from the field type (term vectors, omitNorms, doc values, skip index, points, vectors: :767-768, :771-772, :775-780); payloads follow the MemoryIndex instance (:769), and the ordinal, doc-values generation, attributes (always empty) and the last two flags are not taken from it (:766, :773-774, :781-782).

1. Norms follow. FieldInfo#hasNorms is indexOptions != NONE && omitNorms == false (FieldInfo.java:637-640), and getNormValues returns norms unless omitsNorms() (MemoryIndex.java:2283-2289): a field with no postings gets a norm.
2. The writer's check is correct. addIndexes(CodecReader...) verifies each source FieldInfo before copying (IndexWriter.java:3288-3294 -> FieldInfos.java:401-409, :511-515 -> FieldInfo.java:276-288). SlowCodecReaderWrapper "Wraps arbitrary readers for merging" (SlowCodecReaderWrapper.java:38), and the MemoryIndex reader reports Version.LATEST metadata (MemoryIndex.java:2292-2294), so the copy reaches the schema check, which is fed the wrong FieldInfo.

terms("f") is null on both sides for every non-indexed kind. The hunt's matrix found the same for all five doc-values types (numeric, sorted and sorted-numeric also with a skip index), IntPoint 1D and 2D, LongField, float and byte vectors and StoredField string, int and binary; the other eight reader kinds it tested showed no exception and no mismatch (only an informational FieldInfo difference on the soft-deletes reader's own soft-deletes field). IntField comes from the devil's-advocate verifier's probe.

The FieldInfo is per field name, not per added instance. A name that also has an instance that indexes terms really has terms and is reported correctly: a TextField and a NumericDocValuesField both named f, in either order, give DOCS_AND_FREQS_AND_POSITIONS with norms in both readers. Only a name whose instances all index nothing gets the wrong metadata.

• Into an index holding the same document: IllegalArgumentException: cannot change field "price" from index options=NONE to inconsistent index options=DOCS_AND_FREQS_AND_POSITIONS.
• Into an empty index, then addDocument of the same document in a new writer: IllegalArgumentException: cannot change field "price" from index options=DOCS_AND_FREQS_AND_POSITIONS to inconsistent index options=NONE.

• Who is exposed. Applications that build a MemoryIndex from ordinary Documents with doc-values, point, vector or stored-only fields and then copy its reader into a persistent index with addIndexes(SlowCodecReaderWrapper.wrap(reader)), read its FieldInfos (getIndexOptions, hasNorms, getNormValues, FieldInfos.getIndexedFields), or run FieldExistsQuery on a stored-only or point-only field against it. Default IndexWriterConfig, codec and field classes; the one-field repro is enough, and no dependence on document size is apparent from the code.
• Effect. On 9.0.0 and later, a clean IllegalArgumentException when the target already holds the field natively. Into an empty target, a committed segment that records the field as indexed with positions and norms; after it, ordinary addDocument calls with that field are rejected, in this and later writer sessions. That lasts until the copied documents are deleted before they are merged with other documents and a new writer is opened, or the index is rebuilt: deleting them in the same writer session does not help, and once they have been merged with other documents the wrong schema stays after deleting them and forceMerge. Not data corruption: the persisted segment is self-consistent and CheckIndex (level 3) is clean after copying this bug's documents (the CheckIndex failures in the hunt's matrix come from term-vector fields, hunt row LFV-788). On 6.0.0 to 8.11.4 both steps are accepted silently and the wrong schema is persisted. FieldExistsQuery on a stored-only or point-only field counts 1 on the MemoryIndex where a real segment throws IllegalStateException (it never matches a document without the field).
• Scale. One field in one document suffices. In the hunt's matrix of 1,024 one- and two-field documents, copying a default MemoryIndex into an index holding the same document failed on a non-indexed field for 663 of them, in each merge mode (578 with MemoryIndex(true, true), where some fail first on a text field's offsets).
• Limits to state plainly. The same copy also fails, by MemoryIndex's deliberate design and unchanged by the fix, for StringField and KeywordField (every indexed field gets positions, so DOCS becomes DOCS_AND_FREQS_AND_POSITIONS) and for every document of a MemoryIndex(storeOffsets=true) (text fields get offsets). The executed clean case is the default MemoryIndex(false, false) with positional text fields (the verifier ran only (false, false) and (true, true)); this bug blocks such documents when they also carry doc values, points, vectors or stored-only fields. So the strong claim is the reader's FieldInfo/norms contract; the addIndexes failures are a limited consequence (the devil's-advocate verifier rates the reader defect high confidence and the addIndexes framing medium).
• Likelihood: rare. Every MemoryIndex with such a field name has the wrong metadata for it, but it shows only through these compositions. No matching public report was identified in the documented searches, although the code has been this way since 6.0.0 (2016).
• In-tree consumers. lucene/monitor matches a single document through a MemoryIndex(true, true) (DocumentBatch.java:107-113) and a batch of several documents through an IndexWriter (:55-62, :71-74). Its presearcher walks the reader's FieldInfos but skips fields whose terms() is null (TermFilteredPresearcher.java:103-108), so the wrong index options do not change which queries it selects.

The test sits in the MemoryIndex package by convention; it uses no package-private access. Nothing random reaches the code path: it failed with every seed tried (DEADBEEF through gradle with assertions on and off; DEADBEEF, 5EED1234, 5EED3003 and 5EED3004 on a standalone runner with -ea and -da).

Suggested regression test: this same test, for example as a method of TestMemoryIndex.

The addIndexes consequence is not part of the repro. Adding a TextField title to the document and copying the reader with addIndexes(SlowCodecReaderWrapper.wrap(reader)) into an index that holds the same document throws cannot change field "price" from index options=NONE to inconsistent index options=DOCS_AND_FREQS_AND_POSITIONS, and copying into an empty index makes a later addDocument of the same document throw the reverse message (main 645d4fe14a20 and 10.6.0).

TestMemoryIndexDocValuesOnlyFieldInfo > testDocValuesOnlyFieldIsNotIndexed FAILED
    java.lang.AssertionError: index options of "price" expected:<NONE> but was:<DOCS_AND_FREQS_AND_POSITIONS>
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:5261ED1D894D8F3B]:0)
        at junit@4.13.2/org.junit.Assert.fail(Assert.java:89)
        at junit@4.13.2/org.junit.Assert.failNotEquals(Assert.java:835)
        at junit@4.13.2/org.junit.Assert.assertEquals(Assert.java:120)
        at org.apache.lucene.index.memory.TestMemoryIndexDocValuesOnlyFieldInfo.testDocValuesOnlyFieldIsNotIndexed(TestMemoryIndexDocValuesOnlyFieldInfo.java:36)
:lucene:memory:test (FAILURE): 1 test, 1  failure

# the same with -Ptests.asserts=false, on the standalone runner with -ea and -da,
# and on the 10.6.0 and 10.5.2 release jars; with the recommended fix: passes

• Contract. FieldInfo#getIndexOptions: "Returns IndexOptions for the field, or IndexOptions.NONE if the field is not indexed" (FieldInfo.java:564); IndexOptions.NONE: "Not indexed" (IndexOptions.java:29); hasNorms: "Returns true if this field actually has any norms" (FieldInfo.java:637); FieldInfos.getIndexedFields: "fields that have a terms index" (FieldInfos.java:240); LeafReader#getNormValues: "or null if no NumericDocValues were indexed" (LeafReader.java:202-206).
• The other field-type settings already follow the field type. createFieldInfo copies term vectors, omitNorms, doc values, skip index, points and vectors from the FieldType (MemoryIndex.java:767-768, :771-772, :775-780); payloads follow the MemoryIndex instance flag (:769), and the ordinal, doc-values generation, attributes (always empty) and the soft-deletes and parent flags are not field-type settings (:766, :773-774, :781-782). LUCENE-7679 (#8730) moved the code there to re-use as many FieldType settings as possible and kept only the index options as they were.
• History. The expression comes from LUCENE-4515 (59476c30cf3, 2012), when every MemoryIndex field had a token stream: up to 5.5.x MemoryIndex rejected doc-values fields and failed on stored-only fields (null token stream); point fields did not exist before 6.0.0. LUCENE-7091 (#8147, doc values) and LUCENE-7093 (#8149, points), both in 6.0.0, started accepting fields without terms and gave them the same options; neither discussion mentions index options. Stored-only fields were accepted from 6.0.0 only as a side effect of their null token stream; MemoryIndex has kept stored values only since PR #11999 (first release 9.5.0), so before 9.5.0 such a field added only the wrong FieldInfo and no value.
• Precedent. Wrong MemoryIndex FieldInfo bits were fixed as bugs before: LUCENE-6402 (#7462, "MemoryIndex should pass TestUtil.checkReader", the term-vector bit) and LUCENE-7679 (#8730, omitNorms). In PR #907 (LUCENE-10357, closed unmerged) GitHub user jpountz wrote that MemoryIndex should return null Terms only when the field does not exist or its index options are NONE.
• No upstream test failed with the fix. With fix A2 (and, in the first brief, with fix A) the lucene/memory, lucene/monitor and lucene/highlighter suites pass; this shows only that no tested code depends on the current value.

Strongest case (the verifiers and the adjudication): MemoryIndex deliberately uses one set of index options per instance. Every indexed field gets positions, keyword fields get fake ones (#12139), and TestMemoryIndex#testKeywordWithoutTokenStream asserts freq and positions for DOCS and DOCS_AND_FREQS field types. MemoryIndex is a single-document search helper that never promised IndexWriter-identical FieldInfos or support as an addIndexes source; FieldInfo APIs are expert; and an indexed field without terms is legal ("ghost fields", LUCENE-10357).

Why it fails: that design covers fields that do index terms, for which MemoryIndex really stores (fake) positions, so their postings match their FieldInfo. For a field whose type says NONE, MemoryIndex stores nothing, and the FieldInfo constructor already normalizes non-indexed fields ("for non-indexed fields, leave defaults", FieldInfo.java:110-114), so reporting NONE does not clash with per-instance payloads. Ghost fields come from merges that drop every value of a field that was indexed; they do not make a never-indexed field indexed.

The recommended fix is the finder's three-part sketch plus one guard in addField, applied to 645d4fe14a20 in a scratch worktree and run through gradle. The sketch alone, two partial variants and two alternatives are listed with why they fall short.

Four small changes in MemoryIndex.java (one file, +56/-24, most of it the re-indented addField block): createFieldInfo uses IndexOptions.NONE when fieldType.indexOptions() is NONE, else the old expression (now indexedOptions()); getInfo rebuilds the FieldInfo (its index options are final) with indexedOptions() and that instance's storeTermVectors/omitNorms when a name first added without terms gets an instance that indexes terms (Info.fieldInfo is not final, :1223, so it can be replaced); getNormValues returns null unless fieldInfo.hasNorms(); and addField(IndexableField, Analyzer) runs its existing term-storing block (:624-645) only when the field type's index options are not NONE, as IndexWriter does (IndexingChain.java:1481). The inner else if (... subsumes(IndexOptions.DOCS)) becomes else.

// addField(IndexableField, Analyzer): like IndexWriter, invert only fields whose type indexes something
if (field.fieldType().indexOptions() != IndexOptions.NONE) {
  ... // the existing token-stream / binary-value block; its "else if (... subsumes(DOCS))" becomes "else"
}
// createFieldInfo:
IndexOptions indexOptions =
    fieldType.indexOptions() == IndexOptions.NONE ? IndexOptions.NONE : indexedOptions();
// getInfo: rebuild the FieldInfo with indexedOptions() when a name first added without
// terms gets an instance that indexes terms (FieldInfo's index options are final)
// getNormValues:
if (info == null || info.fieldInfo.hasNorms() == false) {
  return null;
}

Trade-offs

• All parts are needed: createFieldInfo alone would report a name first added as a doc value and then as a TextField as NONE although it has terms, because getInfo builds the FieldInfo once; norms must follow hasNorms(); and without the addField guard a custom field whose type says NONE but whose token stream emits terms would be reported NONE while its terms stay searchable (option A).
• Keeps MemoryIndex's deliberate design for fields that index terms (positions everywhere, offsets with storeOffsets), so StringField/KeywordField and storeOffsets copies still fail as before.
• Behaviour change: FieldExistsQuery on a stored-only or point-only MemoryIndex field throws IllegalStateException, as on a real segment, instead of matching (executed with A2 for StoredField and IntPoint with assertions on, and with fix A).
• Behaviour change: a custom field whose type says NONE but whose token stream emits terms is no longer searchable in a MemoryIndex. It never was in an IndexWriter index, and lucene/monitor matches a single document with a MemoryIndex and several with an IndexWriter (DocumentBatch.java:55-62, :71-74, :107-113), so the two batch kinds then agree. Stock field classes never produce such a field (Field.java:466-468). A2 does more than drop the terms: its guard sits before the token-stream call, so addField no longer calls tokenStream() for any field whose type says NONE, as IndexWriter (IndexingChain.java:1481). A custom IndexableField whose tokenStream() has side effects (validation that throws, resources it opens or closes) no longer sees them run in a MemoryIndex; whether anyone relies on that is unknown.
• Test gap: the custom-field case is shown only by standalone probes; neither the suggested regression test nor any gradle suite exercises it. Proposing A2 rather than A likely needs a focused test of a custom NONE field with a token stream, and agreement that such a field should not be searchable.
• The getInfo part also takes omitNorms from the first instance that indexes terms, which overlaps the first-instance case left by #8730 (LUCENE-7679).

What the checks validated

• Gradle, fresh scratch worktree of 645d4fe14a20, seed DEADBEEF: stock + test fails; with A2 the whole :lucene:memory:test passes, 4 suites, 52 tests (the repro, TestMemoryIndex 30, TestMemoryIndexAgainstDirectory 16, TestSlicedIntBlockPool 5), 0 failures, with assertions on and off; lucene/monitor (32 suites, 181 tests, 1 skipped) and lucene/highlighter (34 suites, 722 tests) pass with assertions on; :lucene:memory:check -x test passes (google-java-format, ecjLint, forbiddenApis).
• Standalone runner with the A2 class shadowed: the repro passes; the finder's four tests pass (both addIndexes directions succeed); a field-name probe matches IndexWriter in all four cases, including the custom NONE field with a token stream (NONE, no norms, no terms, 0 matches).
• Not run: the full Lucene test suite, nightly tests, benchmarks.

The first three parts of A2 only (one file, +33/-4): createFieldInfo takes NONE from the field type, getInfo upgrades on the first instance that indexes terms, getNormValues follows hasNorms().

Trade-offs

• Smaller and enough for every stock field class.
• Gap: addField stores the terms of any non-null token stream whatever fieldType.indexOptions() says (MemoryIndex.java:636-637), while IndexWriter inverts a field only when its index options are not NONE (IndexingChain.java:1481). For a custom IndexableField whose type says NONE and whose tokenStream returns tokens (executed): IndexWriter reports NONE and no terms; stock MemoryIndex reports positions, norms and searchable terms; with fix A MemoryIndex reports NONE without norms but still serves the terms (terms("f") non-null, a TermQuery matches), so its reader disagrees with itself. A2 closes this.

What the checks validated

• In the first brief (gradle, scratch worktree of 645d4fe14a20): the whole lucene/memory suite, 52 tests, assertions on and off; lucene/monitor 181 tests (1 skipped) and lucene/highlighter 722 tests, 0 failures; :lucene:memory:check -x test passes; the finder's four tests pass on the standalone runner.
• In the hunt (standalone runner, a9779ef3c193): the finder's 4 tests, the independent reproduction's 8 tests, the cross-model reviewers' own tests and the intended-behaviour check's probe pass with it.

Use NONE when the field type indexes nothing, nothing else.

Trade-offs

Mis-reports a name whose first instance has no terms and a later one has (read from getInfo, not run); leaves the norms keyed on omitsNorms().

Return null unless hasNorms().

Trade-offs

Removes nothing as long as the index options stay hard-coded (hasNorms() stays true); the addIndexes failures remain (read, not run).

Full IndexWriter parity, also for fields that index terms.

Trade-offs

Would also fix the StringField/KeywordField copies, but contradicts MemoryIndex's deliberate positions-for-every-indexed-field design that TestMemoryIndex#testKeywordWithoutTokenStream asserts and #12139 built on (read, not run).

Say in the javadoc that the reader is not an addIndexes source and its metadata is approximate.

Trade-offs

Leaves the FieldInfo#getIndexOptions contract broken for every consumer of the reader.

Method: semi-formal (bounded-exhaustive translation validation of the real jar), a hunt unit, 2026-10-27. The harness copied 32 field kinds as single-field documents and all 992 ordered pairs (1,024 documents) from 10 reader kinds (SegmentReader, one with a deleted document, MemoryIndex default and with offsets and payloads, ParallelLeafReader, FilterLeafReader, an ExitableDirectoryReader leaf, a SoftDeletesDirectoryReaderWrapper leaf, SortingCodecReader, FilterCodecReader), with a plain and a reordering merge policy, into three targets (empty then a native addDocument and forceMerge, holding the same document, holding it with the fields reversed). Oracles: no exception, equal content dumps, CheckIndex with slow checks. 20,480 cases and 123,410 checks with -ea on a9779ef3c193; replicated with 36 field kinds and a deliberately broken mutant reader with -da (28,512 cases, 174,514 checks; the mutant was caught).

Exceptions came only from the two MemoryIndex kinds. Besides this bug, the matrix also surfaced two related, already known MemoryIndex FieldInfo problems at the same code (term vectors without the FieldInfo flag, #7462 family, and settings taken from the first instance of a name, #8730), tracked as separate hunt rows (LFV-788 and LFV-342). The unit started from an executed lead of an earlier unit.