PerFieldPostingsFormat merge keeps stale format attributes on a deleted-away field after a non-per-field postings segment, so the merged segment cannot be opened

PerFieldPostingsFormat records, per indexed field of a segment, which postings format and file group (suffix) hold the field's postings, as two FieldInfo attributes, and the reader opens exactly those groups. Since LUCENE-8601 (7.7.0) merged FieldInfos inherit the source attributes, and the per-field postings merge rewrites them only for the fields that the source postings readers list. A segment written by a codec whose postings format is not per-field (in-tree SimpleTextCodec, the test framework's CheapBastardCodec, or any custom codec with a plain postings format) does not list a field whose documents were all deleted, so that field keeps its inherited format and suffix; a custom merge-reader wrapper (FilterCodecReader through addIndexes or OneMerge.wrapForMerge) that hides an indexed field's postings does the same with the per-field codec in every session. When the next per-field merge writes no group with that pair (the field's format has a name no merged field uses, or no indexed field is left), the committed segment names postings files that do not exist: DirectoryReader.open and NRT open throw CorruptIndexException and CheckIndex reports the segment broken, while every writer call returned normally and an open writer keeps committing on top. Every executed release from 7.7.0 to 10.5.2 and main is affected; default configurations are not. The fix takes the merge's field list from mergeFieldInfos, as the doc-values and vectors merges already do.

• Per-field postings: the default codec's postings format is a PerFieldPostingsFormat. It lets each field use its own postings format (getPostingsFormatForField, the documented way to customise postings) and writes each format's fields into a file group named <segment>_<format>_<suffix>.*, for example _4_Lucene104_0.psm or _4_FST50_0.psm.
• The note: two FieldInfo attributes per indexed field, PerFieldPostingsFormat.format and PerFieldPostingsFormat.suffix. On reading, they are the only routing record: the reader opens the group of every indexed field that has them (PerFieldPostingsFormat.java:295-321).
• Non-per-field postings codec: a codec whose postingsFormat() is a plain format. In-tree: SimpleTextCodec (lucene/codecs, "FOR RECREATIONAL USE ONLY") and the test framework's CheapBastardCodec; otherwise a custom codec. The codec can differ per writer session (IndexWriterConfig.setCodec).

1. On merge, the merged FieldInfos copy every source field's attributes (SegmentMerger.java:232-239; FieldInfos.java:775-777 and :801; documented at :731-732 and :756-757).
2. The per-field postings merge builds its field list from the source postings readers' iterators only (PerFieldPostingsFormat.java:171-181) and writes fresh attributes only for those fields (:248-253; TODO at :210: "This should check current format from the field attribute?").
3. A per-field source lists every field that has the note (:318, :331-332). A plain source lists only the fields it wrote postings for. In the probe on a9779ef3c193, the SimpleText segment lists [id] while its FieldInfos hold [id, tag], with tag = FST50/0. MergeState takes a source's FieldInfos and its postings reader separately (MergeState.java:133, :155), so a reader wrapper whose getPostingsReader() leaves out an indexed field has the same effect (executed, see Who is affected).
4. So tag keeps FST50/0 through the final merge, which writes only _4_Lucene104_0.*. The reader then tries _4_FST50_0.psm and fails (the comment at :305: "null formatName means the field is in fieldInfos, but has no postings!").
5. Necessary condition: the stale pair is not a group the final merge writes. That holds when the field's format has a name no field of the final merge uses (FST50 here), or, with the default codec alone, when no indexed field is left to merge. For a second instance of a format name that another field also uses, it depends on the suffix order: groups are keyed by format instance, suffixes are counted per format name in field order (:202-230). When the pair coincides with a written group, the segment opens and the field reads as absent.
6. Doc values and vectors are not affected: their merges iterate mergeState.mergeFieldInfos and rewrite every field (PerFieldDocValuesFormat.java:135-140, KnnVectorsWriter.java:135).

Segment _0 (session 1): id = Lucene104/0, tag = FST50/0, files _0_Lucene104_0.* and _0_FST50_0.*. Segment _2 (SimpleText, session 2): 2 documents, terms(tag) null, attributes unchanged. Segment _4 (session 3): 3 documents, postings files _4_Lucene104_0.* only, tag = FST50/0. DirectoryReader.open: CorruptIndexException caused by FileNotFoundException: No sub-file with id _FST50_0.psm found in compound file "_4.cfs". CheckIndex (which reports, it does not throw): 1 broken segment containing 3 documents. The session-2 segment must keep a live document: a fully deleted segment is dropped with its notes.

• Who is hit. Applications that mix writer sessions of a per-field codec with sessions of a codec whose postings format is not per-field. A uniquely named, SPI-registered custom codec with a plain Lucene104PostingsFormat triggers it too, in a public-API main() on FSDirectory without SimpleText. lucene/benchmark's CreateIndexTask can install a FilterCodec with any named plain postings format (codec.postingsFormat, CreateIndexTask.java:150-161), a benchmark setting. Solr ships a SimpleTextCodecFactory an operator can select, documented as never for production (read, not executed); the shipped codecs of two other common search servers build per-field postings (read).
• A second route: merge-source reader wrappers. OneMerge.wrapForMerge exists to "Wrap a reader prior to merging in order to add/remove fields or documents" (MergePolicy.java:302), and addIndexes(CodecReader...) takes any CodecReader (IndexWriter.java:3275). A FilterCodecReader whose getPostingsReader() leaves out an indexed field but keeps its FieldInfo reaches the same stale note with the per-field codec in every session, no deletion and no SimpleText needed: 2 of 2 fail (one test per route) on main with -ea and -da and on 10.5.2, the same wrapper hiding nothing passes, and the fix passes. In-tree wrappers do not hide postings this way (SlowCodecReaderWrapper lists every indexed FieldInfo, SortingCodecReader passes the field list through).
• Opt-in, not default. With a per-field codec in every session and no wrapper that hides postings it cannot happen (by reading: a per-field source lists every field that has a note). With the default codec plus a SimpleText session it needs the "no indexed field left" case.
• The other ingredients of the codec route are ordinary. Deleting every document that has some field, then merges. Deterministic with 4 documents (the independent reproduction: 20 of 20 random seeds; this doc: 30 of 30).
• What is lost. The whole merged segment, not just the field: DirectoryReader.open, NRT open and openIfChanged throw; an open writer is not stopped and keeps committing, so under the default KeepOnlyLastCommitDeletionPolicy no older readable commit is left; a new writer that must merge the segment closes with a tragic CorruptIndexException; CheckIndex reports it and -exorcise drops all of its documents (3 of 4 in the security probe on main ea57360912ed). Recovery: reindex or exorcise, unless an older commit (for example kept by a snapshot deletion policy) or a backup exists (not tested).
• Fails closed. No hang, no wrong hits, no other field's data, no deleted documents coming back (security probe, -ea and -da).
• Unreported since 7.7.0 (2019), consistent with the narrow trigger.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => org.apache.lucene.index.CorruptIndexException: Problem reading index from ByteBuffersDirectory@1999b25c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@405e4620 (resource=ByteBuffersDirectory@1999b25c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@405e4620)
       org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:142)
       org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:95)
       org.apache.lucene.index.StandardDirectoryReader.createOrReuseSegmentReader(StandardDirectoryReader.java:296)
       org.apache.lucene.index.StandardDirectoryReader.createSegmentReaders(StandardDirectoryReader.java:239)
       [...]
     Caused by: java.io.FileNotFoundException: No sub-file with id _Lucene104_0.psm found in compound file "_3.cfs" (fileName=_3_Lucene104_0.psm files: [.fnm, .fdx, .fdt, .fdm])
       org.apache.lucene.codecs.lucene90.Lucene90CompoundReader.openInput(Lucene90CompoundReader.java:184)
       org.apache.lucene.store.Directory.openChecksumInput(Directory.java:156)
       org.apache.lucene.codecs.lucene104.Lucene104PostingsReader.<init>(Lucene104PostingsReader.java:85)
       org.apache.lucene.codecs.lucene104.Lucene104PostingsFormat.fieldsProducer(Lucene104PostingsFormat.java:423)
       org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:316)
       [...]

• The contract. getPostingsFormatForField: "The field to format mapping is written to the index, so this method is only invoked when writing, not when reading." (PerFieldPostingsFormat.java:379-385). The attributes are the reader's only routing record.
• Codecs may differ per session. IndexWriterConfig.setCodec: "Only takes effect when IndexWriter is first created." (IndexWriterConfig.java:292-297). TestMixedCodecs sets SimpleText on random writer sessions ("Make sure we aggressively mix in SimpleText since it has different impls for all codec formats", TestMixedCodecs.java:49-52), but its only field never loses all of its documents. The test framework picks SimpleTextCodec or CheapBastardCodec as a test's random default codec (SetupAndRestoreStaticEnv.java:185-197).
• Nothing requires per-field postings. The codecs package documentation describes writing a full custom codec as well as extending the default codec via getPostingsFormatForField (codecs/package-info.java:40-64).
• The sibling merges handle this case. Doc values and vectors iterate mergeFieldInfos; LUCENE-9992 (#11031, a deleted-away vector field kept its FieldInfo with no vector files) and LUCENE-10228 (PR #432, an inherited per-field suffix in the vectors merge) were fixed as bugs, for vectors only.
• History. Before 7.7.0, merged FieldInfos started with empty attributes, so such a field had no note and the reader skipped it. LUCENE-8601 made them inherit attributes and turned the per-field formats' "found existing value" checks into overwrites; its thread (#9647) says the formats "expect it to be null and generate new values during merge" and does not discuss fields the postings merge never visits (our reading).
• No test, CHANGES or MIGRATE entry, javadoc or discussion accepts the behaviour (the intended-behaviour check's 40 intent queries; the follow-up's 115 queries).

The strongest case for "intended or unsupported": SimpleTextCodec is "FOR RECREATIONAL USE ONLY" and @lucene.experimental, so mixing it into a production index is unsupported; PerFieldPostingsFormat is itself @lucene.experimental (PerFieldPostingsFormat.java:63), its two attributes are private codec bookkeeping, and nothing promises that any codecs can be alternated or that a wrapper's postings may disagree with its FieldInfos; FieldInfo.putAttribute says that "If the value of the attributes for a same field is changed between the documents, the behaviour after merge is undefined" (FieldInfo.java:669-671); and the finder's first repro uses an anonymous FilterCodec that borrows the default codec's name, although FilterCodec asks for "a unique name" (FilterCodec.java:53).

Why it fails: the recreational and experimental labels bound the support promise, not the outcome: every writer call succeeds and commits a segment that the same codec cannot read, and the reader already has a representation for an indexed field without postings, no note (:305), which the merge could write. SimpleText is not needed: the test framework's CheapBastard codec and a uniquely named custom codec with a plain postings format fail the same way. The putAttribute sentence covers conflicting values for one field; here the value never changes and only one source segment carries the field. This doc's test uses a Lucene104Codec subclass, the documented extension point, with no name trick. Every verifier rejected the intended reading; the intended-behaviour check and the cross-model reviews rated it weak (the independent reproduction gave no rating), and a later cross-model fact check of this doc rated it moderate and still judged it a bug. What partly survives is reach: a maintainer may call it low priority.

One fix was built and validated on a scratch worktree of main a9779ef3c193; three alternatives are listed with their trade-offs.

Workaround until fixed: keep every writer session of a per-field index on a per-field postings codec, and do not wrap merge sources so that their postings leave out an indexed field. No workaround was tested for an index that already holds a plain-postings segment: any later per-field merge of it can hit the bug, also with the default codec alone. Reindexing with a per-field codec avoids it; CheckIndex only detects it after the merge was committed.

In PerFieldPostingsFormat.FieldsWriter.merge, replace the MergedIterator over the source readers' iterators (PerFieldPostingsFormat.java:171-181) with the merged segment's indexed fields, so buildFieldsGroupMapping writes a current format and suffix for every one of them. This is the rule PerFieldDocValuesFormat.merge already follows (:135-140). The now-unused imports Arrays, Objects and MergedIterator go.

// Assign a format to every indexed field of the merged segment, not only to the fields the
// source postings readers list: the merged FieldInfos inherit the source attributes, so a
// field left out here (for example one whose postings a non-per-field format dropped) would
// keep attributes that name files this merge never writes. Sorted for a stable suffix order.
List<String> indexedFieldNames = new ArrayList<>();
for (FieldInfo fi : mergeState.mergeFieldInfos) {
  if (fi.getIndexOptions() != IndexOptions.NONE) {
    indexedFieldNames.add(fi.name);
  }
}
indexedFieldNames.sort(null);
Map<PostingsFormat, FieldsGroup> formatToGroups = buildFieldsGroupMapping(indexedFieldNames);

Trade-offs

• A field with no postings left joins its format's group, which is written (the independent reproduction's fix check: _4_FST50_0.* exists, CheckIndex clean); the field reads as absent. A format whose only fields were deleted away therefore gets a file group with no terms: a few small files. That a group with no terms is written and readable was shown for FST50 and Lucene104; a custom postings format would need the same check.
• Suffix numbers can differ from today's when such a field sorts between others; deterministic either way, and every field's attributes are rewritten.
• Cost: one pass over the merged FieldInfos. Not benchmarked.

What the checks validated

• Gradle, -Ptests.seed=DEADBEEF: the new test 3/3 with assertions on and off, 30/30 over 10 random-seed iterations; the closest suites (codecs.perfield.*, TestMixedCodecs, TestAddIndexes, TestSegmentMerger, TestIndexWriterMerging, TestIndexWriterForceMerge, TestFieldInfos) 317 tests, 0 failures (17 skipped), assertions on and off.
• With assertions on only: the whole :lucene:core:test suite with the new test, 8,892 tests, 0 failures (322 skipped); :lucene:codecs:test 693 tests (78 skipped) and :lucene:backward-codecs:test 2,339 tests (184 skipped), 0 failures; :lucene:core:check -x test (google-java-format, ecjLint, forbiddenApis) passes.
• Repro harness with the fix shadowed on a9779ef3c193: this test 3/0 (-ea, -da), the finder's repro 2/0, the independent reproduction's Lucene104Codec-subclass test 1/0, the reader-wrapper probe 2/0 (-ea, -da). Earlier, on ea57360912ed: the finder's exhaustive harness T6 and T6b went from 1716 failures each to 0.

Keep the reader-derived field list and remove the two attributes from every other indexed field of the merged segment. This writes the reader's own representation of an indexed field without postings, no note ("null formatName means the field is in fieldInfos, but has no postings!", :305), and creates no empty file groups.

Trade-offs

Does not depend on how each format writes a group with no terms, and a maintainer may well prefer it; but FieldInfo has no way to remove an attribute, so it needs a new internal API. Not run.

Open a group only if its files exist.

Trade-offs

Hides real corruption, and leaves wrong notes in the index. Not recommended. Not run.

Merged FieldInfos start with no codec attributes again, as before 7.7.0.

Trade-offs

7.7.0 with that hunk reverted passes the repro, but the carry-over is intended and documented since 9.0 and user attributes rely on it. Not recommended.

The hunt's hunt unit on per-field formats built a bounded-exhaustive translation-validation harness (TestH58PerFieldExhaustive) on the stock main jars: real IndexWriter and codecs, 3 postings, 3 doc-values and 3 vector fields, 3 format slots per field (two instances of the default format and one with another name) and 8 writer-session templates T1-T7 and T6b (remaps, deletes, addIndexes, doc-values updates, NRT, mixed codecs).

While writing the harness's attribute-bookkeeping model (merge field set = the source postings readers' fields; merged FieldInfos = union of the source attributes), the AI tool saw that a field outside the merge field set keeps an inherited format and suffix. The run then confirmed it: template T6 (per-field, then SimpleText with deletes and a merge, then a per-field merge) failed 1,716 of 10,935 cases, and T6b (the same through addIndexes(CodecReader)) 1,716 of 5,832; every failure was predicted and every predicted case failed; the other 6 templates had 0 failures. The reader-wrapper route was found later, by the cross-model fact check of this doc reading the code, and then executed. No model checker, SMT solver or prover was involved.