Mixing SuggestField and ContextSuggestField under one field name closes the IndexWriter at flush and at every merge that combines the two types

The document suggester has two field classes: SuggestField writes type byte 0 and ContextSuggestField type byte 1 into every completion posting payload. One field name may not mix them; the design (LUCENE-6459) was that a mix is rejected with an IllegalArgumentException "upon indexing". But addDocument never compares the types. The only check is in CompletionFieldsConsumer, while the completion postings are written at flush and at merge, and core treats any exception there as aborting (flush) or tragic (merge). So the IndexWriter closes itself and drops every uncommitted document, unrelated ones included. If the two types landed in different segments, both commit cleanly and CheckIndex passes, but every merge that combines them closes the writer, in every new writer too, until one type has no live documents left. Present since 5.3.0 (2015); executed on 5.3.0 through 10.5.2 and main. A 4-file fix that marks the field as mixed instead of throwing, and lets CompletionQuery reject queries on it as it already does across segments, keeps the writer open; it passes the repro and the whole suggest module.

• Document suggester: SuggestField and ContextSuggestField (lucene/suggest, @lucene.experimental) index suggestions that SuggestIndexSearcher serves from a per-segment FST. The field must be routed to a completion postings format (for example Completion104PostingsFormat) through a per-field codec.
• Type byte: each class writes its type into every posting payload, after the surface form and the weight: 0 for SuggestField (SuggestField.java:64, :123-124, :133), 1 for ContextSuggestField (ContextSuggestField.java:51, :106-107). Both use the same FieldType (SuggestField.java:53-62, :79), so the indexing chain's per-field schema checks cannot tell them apart.
• Aborting and tragic exceptions: an exception while a flush writes a segment aborts it and fires a tragic event; any exception in a merge is tragic. A tragic event rolls the writer back: it closes, and everything since the last commit is gone.

1. The only comparison is at write time. CompletionFieldsConsumer.write writes the delegate postings, then one CompletionTermWriter per field (:85-113). CompletionTermWriter.write reads every posting's type byte; the first sets the field's type and a different one throws IllegalArgumentException("single field name has mixed types") (:214-220, throw at :219).
2. It runs inside flush and merge. DocumentsWriterPerThread.flush catches every Throwable as the aborting exception (:585-589), and maybeAbort aborts and fires the tragic event (:594-605). IndexWriter.merge catches every Throwable and calls tragicEvent(t, "merge") (:5109-5113).
3. Each segment stores one type. The type is saved per field and segment (CompletionFieldsConsumer.java:105-110, :149), so a segment of each type flushes and commits cleanly. A merge re-reads the live postings of its input segments; if they hold both types it hits the check again, in every writer, until one type has no live documents left. Same-type merges work.
4. The query side already copes. CompletionQuery.rewrite compares the segments' types and throws IllegalStateException("... has values of multiple types") (CompletionQuery.java:87-103), added in the same commit as the write-time check.

Commit doc 1 (SuggestField "apple"): segment A, type 0. Commit doc 2 (ContextSuggestField "apricot"): segment B, type 1. Both commits succeed; CheckIndex is clean. Add doc 3 (no suggest field) and call forceMerge(1): the merge sees type 0 then type 1 and throws; the writer closes; reopening the index finds 2 documents. A new writer's forceMerge(1) closes it too. Deleting doc 2 lets the merge succeed.

Also affected: a buffered segment holding both types (the next commit, NRT reopen, or close() with the default commitOnClose=true, closes the writer and loses all uncommitted documents), upstream's own input of one document with both fields, a natural merge picked by the merge policy, and addIndexes(Directory), which imports a segment of the other type with no check. addIndexes(CodecReader) over both types throws but leaves the writer open (the finder's probe and the fact checks executed it).

• Who is exposed. Direct users of SuggestField / ContextSuggestField with a field routed to a completion postings format. Triggers: adding contexts to an existing suggest field without reindexing, two code paths (one with contexts, one without), or one document with both fields (upstream's own test input).
• Buffered mix. The next flush (commit, NRT reopen, close() with the default commitOnClose=true, or an automatic flush) closes the writer and loses every document added since the last commit, unrelated ones included: in the repro the index keeps only the 1 previously committed document.
• Committed mix. Both commits succeed, CheckIndex is clean and suggest queries already fail with the designed IllegalStateException. Every merge whose input holds live documents of both types then closes the writer: forceMerge, a natural merge, any new writer. Each time the uncommitted documents are lost. It ends once one type has no live documents left: delete them, hard-update them to the other class (executed), or reindex.
• Default settings. Beyond the per-field codec the suggester needs anyway, nothing unusual is required: an independent probe closed the writer through a natural merge with TieredMergePolicy and ConcurrentMergeScheduler after 8 alternating commits.
• Bounded. The committed index stays readable and CheckIndex-clean. No scale is needed.
• Soft-update migrations stay stuck. With SoftDeletesRetentionMergePolicy, the retained soft-deleted old version keeps the other type in the merge: after a softUpdateDocument from SuggestField to ContextSuggestField, forceMerge(1) closes the writer, while a hard updateDocument does not (the finder's exploratory probe; re-run by the cross-model fact check's probe with assertions on and off).
• How likely. Rare: both classes under one name, Elasticsearch's completion mapping does not allow changing a field's contexts, and no report found since 5.3.0 (2015).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: the merge closed the writer expected null, but was:<java.lang.IllegalArgumentException: single field name has mixed types>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotNull(Assert.java:756)
       org.junit.Assert.assertNull(Assert.java:738)
       org.apache.lucene.search.suggest.document.TestCompletionMixedTypesMerge.testMergeOfCommittedSuggestAndContextSuggestSegments(TestCompletionMixedTypesMerge.java:49)

• The design chose input validation. In the LUCENE-6459 review (#7518) a reviewer asked "Are you allowed to mix ContextSuggestField and SuggestField even for the same field name, within one suggester?"; the author answered "No you are not. If mixed, CompletionFieldsConsumer will throw IllegalArgumentException upon indexing.", and the reviewer replied "OK, excellent." Nothing in the thread mentions the writer closing, lost documents or merges.
• IndexWriter's contract. The class NOTE: the writer closes itself "if you hit an Error, or disaster strikes during a checkpoint", as "a defensive measure in case any internal state (buffered documents, deletions, reference counts) were corrupted" (IndexWriter.java:162-165). addDocument: on an exception "the index will be consistent, but this document may not have been added" (:1465-1466). Design intent in the package-private DocumentsWriter (:67-79): aborting exceptions are the "certain limited times when an exception can corrupt this state"; "All other exceptions" only mark the document deleted.
• The merge case rejects no input. Both segments were accepted and committed, CheckIndex is clean and the query side reports the state cleanly (CompletionQuery.java:87-103). Failing every merge in every writer is not validation.
• The only test never looks at the writer. TestContextSuggestField#testMixedSuggestFields (:158-176) expects the exception from addDocument or commit(false) and stops there. No test covers two documents, separate segments, merges or addIndexes.
• No user-facing documentation of the rule in the javadoc of SuggestField, ContextSuggestField, CompletionPostingsFormat, CompletionQuery or the package.
• Precedents. LUCENE-5662 (#6724, "Missing Field null checks can result in aborted segments", fixed in 4.8.1) and LUCENE-8441 (#9487, a wrong index-sort field type now fails the document when it is added, 7.5) moved validation out of aborting steps as bug fixes. LUCENE-7562 (#8613) fixed a different merge-time exception in this same file, one that made merges fail again and again, as a Lucene bug. A design precedent, not a fix: #15584 (2026), an enhancement that added a new indexing-time validation, made it non-aborting from the start (IndexingChain.java:2095-2101), and its test says "ensure IndexWriter remains open and doc was skipped" (TestCustomTermFreq.java:125).
• Timeline. LUCENE-6579 (#7637), which made every merge exception tragic, was resolved on 2015-07-14, after the design answer; both shipped in 5.3.0.

The strongest case for "works as designed" (built by the devil's advocate; risk moderate): mixing is invalid by design (the review answer); upstream's test expects an exception from its add-and-commit block, and a 2019 commit (ad457d188e5, "Improve RIW exception handling and opt out of concurrent flushing if exception is expected") adjusted that test for a flush-time exception; core makes every flush or merge exception aborting by design (LUCENE-5987, LUCENE-6579, TestIndexWriterExceptions#testDocumentsWriterAbort); the classes and the codec API are @lucene.experimental, and the codec API cannot reject a document without aborting; the module already calls the cross-segment state invalid at query time; a major downstream server blocks the mix in its mapping.

Why it fails: it explains why an exception is thrown, not why the writer must close and lose unrelated documents; no document, test or discussion accepts that. For the merge case it offers only the generic rules (one schema per field name, every merge exception tragic); it does not say why an index that accepted and committed both segments and passes CheckIndex should close every writer that merges them, while the query side already reports the state as a clean error. The way out is rewriting data. The IndexWriter contract describes self-closing as a guard against corrupted state, upstream fixed this shape as a bug before (LUCENE-5662, LUCENE-8441, LUCENE-7562), and #15584 made a new validation non-aborting by design. Verdict: a defect (adjudication: confirmed, medium confidence). A maintainer may still call the one-segment flush case expected, or file the whole as an improvement (validate earlier, document the rule); the cross-model fact check rates the "works as designed" case strong for that flush case and moderate for the merge.

Only option A was built and validated. It moves the error from indexing time to query time, so upstream's testMixedSuggestFields must change; a maintainer may prefer to keep the rejection at indexing time (option B), which needs a core hook because nothing at addDocument knows the field's type in the rest of the index. Until a fix ships, using ContextSuggestField (with zero contexts where none apply) for every document of the field avoids the mix.

CompletionTermWriter.write records MIXED_TYPE (2) instead of throwing; CompletionQuery.rewrite throws its existing IllegalStateException("... has values of multiple types") for a mixed segment; CompletionTerms.getType() and the format javadoc mention the new value. Merges re-derive the type from live postings, so deleting every document of one type and merging makes the field queryable again (probe executed). testMixedSuggestFields changes from "commit throws" to "a suggest query throws".

--- CompletionFieldsConsumer.java (CompletionTermWriter.write)
           } else if (this.type != type) {
-            throw new IllegalArgumentException("single field name has mixed types");
+            // SuggestField and ContextSuggestField values under one field name. Throwing here would
+            // abort the flush or merge and close the IndexWriter, so mark the field instead and let
+            // CompletionQuery reject queries on it.
+            this.type = CompletionPostingsFormat.MIXED_TYPE;
           }
--- CompletionQuery.java (rewrite)
         byte t = completionTerms.getType();
+        if (t == CompletionPostingsFormat.MIXED_TYPE) {
+          throw new IllegalStateException(getField() + " has values of multiple types");
+        }
--- CompletionPostingsFormat.java: + static final byte MIXED_TYPE = 2; (with javadoc)

Trade-offs

• The error moves from indexing time (today the commit throws and the writer dies) to query time, so users learn about the mix later.
• CompletionTerms.getType() (public, experimental) can return 2 and the .cmp file can store 2. The reader does not validate the byte, so nothing else changes; an older release reading such a segment would run any completion query on the mixed FST without an error. Making old readers refuse it needs a format version bump, and today COMPLETION_CODEC_VERSION is both the minimum and the current version (CompletionPostingsFormat.java:102-103, CompletionFieldsProducer.java:88-89): a plain bump would make the patched reader reject existing segments too, so it needs separate minimum and current values and back-compatibility tests. Not done here.
• addIndexes(CodecReader) over both types, today the one path that rejects cleanly, succeeds and marks the result mixed; a suggest query then throws (executed by the fact checks).
• Soft-update migrations: with SoftDeletesRetentionMergePolicy the merge succeeds (executed), but while the old soft-deleted versions are retained the merge input still holds both types, so the merged segment is marked mixed and suggest queries stay rejected until retention drops them (read, not queried). Upstream would have to decide whether that is acceptable.
• The sibling bug (deleted or rejected postings counted at flush): its 3 tests also pass, but such a segment is marked mixed until a merge drops the deleted posting, so its own live-postings fix is still worth having.
• Not benchmarked; one assignment replaces a throw, and rewrite adds one byte comparison per segment.

What the checks validated

• Scratch worktree of a9779ef3c193, gradle, JDK 25: stock + the new test, 6 of 8 fail with assertions on and off; + the fix and the updated testMixedSuggestFields, :lucene:suggest:check passes (ecjLint, forbiddenApis, checkGoogleJavaFormat; 32 suites, 244 tests, 7 skipped, 0 failures, including the new test 8 of 8 and TestContextSuggestField 6 of 6), and the module's tests pass with assertions off.
• Standalone runner: the new test 8 of 8 with assertions on and off; the finder's 4 tests pass; the fix probe passes 2 of 2.

Keeps the error at indexing time as a non-aborting rejection, as the LUCENE-6459 design answer implies. Not built.

Trade-offs

At addDocument nothing in the suggest module knows the field's type in the rest of the index: both classes use the same FieldType, and FieldInfo attributes are not validated across documents ("the behaviour after merge is undefined", FieldInfo.java:669-671). It needs a per-field schema property that the indexing chain checks, the same check in addIndexes(Directory), and handling for indexes that already mix the types across segments (they would still need option A at merge).

Javadoc on SuggestField and ContextSuggestField: never index both under one field name; doing so closes the IndexWriter at the next flush or merge. Not built.

Trade-offs

No behaviour change: the writer still closes and the committed index still cannot be merged.

Method: code reading and exploratory probes, then bounded-exhaustive model-based testing of the real IndexWriter.

• During planning, an independent reviewer from another model family (OpenAI GPT-5.6) read CompletionFieldsConsumer.java and ranked this lead first; a reading of main then confirmed that the type is checked only while postings are written, after the delegate postings, and that flush treats every Throwable as aborting.
• The finder's exploratory public-API probes gave the first executed signal: a tragic close at flush and at merge.
• The finder then wrote a predictive model of IndexWriter with the completion postings format and compared it, operation by operation, with the real writer over every script of length up to 5 over 12 operations (adds of each type and both, plain and rejected adds, delete, flush, commit, forceMerge(1), NRT reopen, addIndexes(Directory), addIndexes(CodecReader)) and 12 configurations: 3,257,424 runs each with assertions on and off, 0 model mismatches (1 to 5 ConcurrentMergeScheduler timing interleavings per run, where forceMerge rethrows before the merge thread closes the writer, were classified separately; natural merges with merge factor 2 were checked only against the contract); a model mutant was killed with 12,328 mismatches, and with the throw removed every contract-violation class dropped to 0. It was minimised into the JUnit; independent AI tools then reproduced, reviewed and judged it.