SharedTermsUniformSplit merge silently drops the postings of fields that an older segment stored in another postings format or shared-terms group

PerFieldPostingsFormat records in every segment which postings format, and which instance of it, holds each field, and it reads each segment back with that segment's own mapping, so the mapping for new segments may change. On merge, the merged segment's fields are grouped by the current mapping, and each format merges its group from a merge state that is restricted to the group's field names but still returns each old segment's own terms. The experimental SharedTermsUniformSplit format (lucene/codecs) keeps the terms of all fields of its group in one shared block, and its custom merge reads, in each old segment, only the block of the first group field it finds. When the group now holds a field that the old segment stored elsewhere (in another format, or in another SharedTermsUniformSplit instance), that field's old postings are never read. The documents stay, but searches, docFreq, deletes and updates by a term of that field miss them; there is no exception, and CheckIndex reports the index clean. It needs a mapping change on an existing index (or an addIndexes from an index with another mapping) and a field-order condition, so it is rare; it has been in every release since the format was added in 8.3.0. A small fix keeps the fast path only when all group fields of a segment share one block.

• Per-field postings. A codec can give each field its own postings format by overriding getPostingsFormatForField (the default codec routes through PerFieldPostingsFormat). Each segment records the format, and which instance of it (a file-group suffix), for every field, and is read back with its own record (PerFieldPostingsFormat.java:295-321). The javadoc: "The field to format mapping is written to the index, so this method is only invoked when writing, not when reading." (:379-385).
• SharedTermsUniformSplit. An experimental postings format in lucene/codecs (STUniformSplitPostingsFormat, SPI name SharedTermsUniformSplit): "the terms of all fields are stored in the same block file" (STUniformSplitPostingsFormat.java:36). All fields that one instance writes in a segment form a group that shares one block and one dictionary.
• Its custom merge. For efficiency the writer merges the source segments' shared blocks directly instead of field by field (STUniformSplitTermsWriter.java:70-74).

1. On merge, PerFieldPostingsFormat groups the merged segment's fields by the current mapping and gives each format a merge state restricted to its group (PerFieldPostingsFormat.java:171-191, PerFieldMergeState.java:44-72).
2. The restriction filters field names only: for each field it still returns the old segment's own terms, from whatever format that segment used (PerFieldMergeState.java:246-256). One target group can therefore span several groups of an old segment.
3. STUniformSplitTermsWriter.merge walks each old segment's group fields in field-number order and stops at the first one with terms ("Iterate all fields only the get the first Terms instanceof STUniformSplitTerms", :295-296). If those terms are not SharedTermsUniformSplit terms, it falls back to the regular merge (:299-304). Otherwise it reads that field's shared block and breaks (:305-313).
4. That block holds only the fields of one old group (STUniformSplitTermsReader.java:109-127). Group fields the old segment stored in another format or another instance are never read, and nothing checks that every field was.

Two documents, fields a (number 0) and b (number 1). Session 1: a in SharedTermsUniformSplit, b in the default format. Session 2: both in one SharedTermsUniformSplit instance, then forceMerge(1). Before the merge every term has docFreq 1; after it, a:a1, a:a2 and b:b2 still have 1 and b:b1 has 0 (a TermQuery finds 0 hits), the index has 1 segment and 2 live documents, and CheckIndex is clean. The merged segment's field b holds only b2.

Condition: the lowest-numbered group field with terms in the old segment must itself be a SharedTermsUniformSplit field; with b numbered first the fallback runs and nothing is lost. The opposite direction (one group split in two) is safe: the merge skips block fields outside the group (:412-419).

• Who uses the format. It is opt-in, @lucene.experimental and lives in lucene/codecs; it is chosen per field by a codec subclass overriding getPostingsFormatForField (the pattern codecs/package-info.java:59-64 documents) or, by code reading, by a Solr schema postingsFormat. A 2020 dev@ mail by the committer who added it says it "is used in production at a massive scale"; no deployment was checked, and no use was found in two other common search servers.
• What triggers it. An index that already stores a field in SharedTermsUniformSplit, then one of: a mapping change that moves another, higher-numbered field into that group; two SharedTermsUniformSplit instances (two groups) becoming one; addIndexes(CodecReader...), or addIndexes(Directory...) plus a merge, from an index written with another mapping. Then any merge of an old segment: forceMerge and addIndexes(CodecReader...) were executed; background merges take the same merge code (read).
• What does not trigger it. No mapping change; a segment written before the field existed (it has no postings of the field to lose); addIndexes from an index with the same per-field grouping (by reading: each old block then holds the whole group); moving every field of a group into the format at once; the group's lowest-numbered field in the old segment not being a SharedTermsUniformSplit field; an index-sorted merge that interleaves documents (it falls back). An index-sorted index whose segments are already in order still loses postings.
• What is lost. The postings of live documents for the moved field, from each old segment that stored the field outside the block the loop reads in that segment (if any old segment's first group field with terms is not a SharedTermsUniformSplit field, the whole merge falls back and nothing is lost). Stored fields and doc values stay; searches, docFreq, filters, deleteDocuments(Term) and updateDocument(Term, ...) by a term of that field miss those documents. The merged segment cannot give the postings back and CheckIndex reports the index clean. With the default deletion policy the old segment files are deleted at commit, so recovery means reindexing from source, or restoring a pre-merge commit that a deletion policy kept (for example a snapshot) or a backup.
• No scale or timing. Two documents, with assertions on and off. The outcome does not depend on the seed: the test framework randomizes the directory and CheckIndex settings, and the 3 bug tests failed 30 of 30 times over 10 random-seed iterations.
• How likely. Rare: mapping changes on existing indexes are infrequent, the field-order condition must hold, and no report of it since 8.3.0 (2019) was found in the searched public sources. Tests never caught it because RandomCodec never picks a UniformSplit format and the UniformSplit suites use one format for every field (TestUniformSplitPostingFormat.java:37).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: docFreq(b:b1) after forceMerge expected:<1> but was:<0>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:647)
       org.apache.lucene.codecs.uniformsplit.sharedterms.TestSTUniformSplitMergeMappingChange.testMergeTakesFieldFromAnotherFormat(TestSTUniformSplitMergeMappingChange.java:65)

• The merge contract. FieldsConsumer.merge: "Merges in the fields from the readers in mergeState. ... Implementations can override this method for more sophisticated merging" (FieldsConsumer.java:66-71). An override may be faster, not lossy.
• The custom merge is an optimization. The class javadoc: the writer "requires a custom merge ... for efficiency. The regular merge would scan all the fields sequentially ... Whereas the custom merge directly scans the internal shared-terms dictionary of all segments to merge" (STUniformSplitTermsWriter.java:70-74). The regular merge, which the existing fallback reaches, keeps every term (control test).
• Mappings may change between segments. getPostingsFormatForField "Returns the postings format that should be used for writing new segments ... The field to format mapping is written to the index, so this method is only invoked when writing, not when reading." (PerFieldPostingsFormat.java:379-385; the Lucene104Codec hook warns only about back-compat, Lucene104Codec.java:170-180). Upstream tests show per-field segments as supported shapes, but not this transition: TestPerFieldPostingsFormat2.testChangeCodecAndMerge carries the comment "Test that heterogeneous index segments are merged successfully" (:126-131), although both of its writer sessions now use the same mapping, and testSameCodecDifferentInstance (:262) treats two instances of one format as two valid groups. Neither test moves a field between formats or groups.
• The code expects per-field use. The loop's comment expects FieldInfos "filtered by PerFieldMergeState" (STUniformSplitTermsWriter.java:292-293), and the loop already falls back for foreign terms, but only looks at the first field.
• No acceptance found. No test, javadoc, CHANGES or MIGRATE entry, JIRA or list thread found in the searched sources accepts the loss or says a field's postings format must not change; the feature thread calls the custom merge "the only tricky part" (#9798).

The strongest case for "intended or unsupported": the format's javadoc says "the terms of all fields are stored in the same block file" (STUniformSplitPostingsFormat.java:36), so its merge may assume one block per old segment; the format is @lucene.experimental; and changing a field's postings format on a live index is unusual.

Why it fails: no such precondition is stated anywhere, and per-field use is expected by the loop's own comment and by the codecs package docs; the sentence describes how one segment is written, not a promise about older segments. Experimental status allows format and API changes between versions, not silent loss of live postings within one version. Nothing forbids a mapping change, the per-field javadoc records the mapping per segment because it may change, and the addIndexes path loses postings with no change inside the destination writer at all. Verdict: not intended, a defect. Every checker (the independent reproduction, the intended-behaviour check, the adjudication and the OpenAI GPT-5.6 reviews, one of them the cross-model review's) rejected the intended reading; the intended-behaviour check and the reviews, which grade it, call it weak. What survives is reach, which bounds priority.

One fix was built and validated in a scratch worktree of a9779ef3c193 (option A, written by the finder; an independent checker wrote an equivalent patch). The other options are untested alternatives.

In STUniformSplitTermsWriter.merge (:281-321), drop the early break: look at every group field of each segment, and fall back to the regular FieldsConsumer.merge as soon as a field's terms are not SharedTermsUniformSplit terms or belong to another shared-terms block than the segment's first such field. One shared-terms reader creates one unionFieldMetadata and hands it to every field of its block (STUniformSplitTermsReader.java:109-127), so an identity check tells blocks apart.

STUniformSplitTerms segmentSharedTerms = null;
for (FieldInfo fieldInfo : mergeState.fieldInfos[segmentIndex]) {
  Terms terms = fieldsProducer.terms(fieldInfo.name);
  if (terms != null) {
    if (!(terms instanceof STUniformSplitTerms sharedTerms)
        || (segmentSharedTerms != null
            && sharedTerms.unionFieldMetadata != segmentSharedTerms.unionFieldMetadata)) {
      super.merge(mergeState, normsProducer); // another format or another block
      return;
    }
    if (segmentSharedTerms == null) segmentSharedTerms = sharedTerms;
  }
}
// then add segmentSharedTerms.createMergingBlockReader() once, as before

Trade-offs

• One mixed old segment sends the whole merge (every segment and every group field) through the regular merge, which the class javadoc calls slower for many fields (it scans the shared dictionary once per field). The merged segment holds the group in one block, so merges take the fast path again once no old mixed segment is left in them and the mapping does not change again.
• Merges without a mapping change keep the fast path; the loop now visits every group field of a segment instead of stopping at the first. Not benchmarked.
• Open PR #16851 edits other methods of the same file; a fix PR may need a rebase.

What the checks validated

• Gradle, -Ptests.seed=DEADBEEF: the new test 4/4 with assertions on and off, 40/40 over 10 random-seed iterations; the closest suites org.apache.lucene.codecs.uniformsplit.* (8 classes with the new test) 86 tests, 0 failures, 4 skipped, assertions on and off (stock: the same 86 with the new test's 3 failures).
• The hunt's other JUnit repros (the finder's, the independent reproduction's, the cross-model review's and the intended-behaviour check's, 17 tests) 0 failures; the whole :lucene:codecs:test suite with the new test: 697 tests, 0 failures (78 skipped); :lucene:codecs:check -x test (google-java-format, ecjLint, forbiddenApis, license checks) passes; tidy leaves the patch unchanged.
• Repro harness on a9779ef3c193 with the fix shadowed: the new test 4/0 with -ea and -da. Earlier, on ea57360912ed: the 729-pair check went from 176 lossy pairs to 0 (that run's JUnit still reports a failure, on purpose: its predictor encodes the stock defect and still predicts the 176), and the fail-open probe passes 3/3.

Add one merging block reader per distinct unionFieldMetadata in the segment, and fall back only for terms of another format.

Trade-offs

Keeps the fast path when two shared-terms groups become one. Needs a check that the merge handles two block readers of one segment (their fields are disjoint). Not run.

Skip the custom merge whenever the merge state is restricted by PerFieldMergeState.

Trade-offs

Simplest, but the format is normally reached through PerFieldPostingsFormat, so this removes the fast path the class javadoc justifies. Not recommended; not run.

Never hand a delegate a group whose fields span several groups of one source segment.

Trade-offs

Larger, and it changes every per-field format's merge for the only postings consumer besides PerFieldPostingsFormat that overrides merge. Not run.

Method: code reading, then bounded-exhaustive translation validation (the hunt's "semi-formal" tier: a hypothesis formed while modelling and witnessed by a tool; no model checker, SMT solver or prover).

• The hunting AI tool of the hunt's unit on per-field formats was building a per-field routing harness and read which postings writers override FieldsConsumer.merge and how they read the restricted merge state. The SharedTermsUniformSplit loop that takes the first field and breaks stood out.
• The harness ran the real IndexWriter and codecs on the stock main jars: fields f0 to f2, each mapped to one of {two SharedTermsUniformSplit instances, the default format} in each of two writer sessions, one document per session, all 27 x 27 = 729 mapping pairs. Oracle: every term of both documents has docFreq 1 after forceMerge(1).
• Result: 176 of 729 pairs lose postings; a predictor written from the root cause gives the exact lost-field set in 729 of 729; with the fix 0 of 729 lose postings. A separate probe showed the loss is silent (CheckIndex clean, stored fields and doc values intact).
• Then a deterministic public-API JUnit; independent checkers reproduced it with their own tests (section "Independent checks").