A DrillDownTermsIndexing.NONE facet dimension in the shared index field makes IndexWriter reject documents since Lucene 9.0

FacetsConfig lets you switch off drill-down terms per dimension with setDrillDownTermsIndexing(dim, DrillDownTermsIndexing.NONE), and by default every dimension writes to the same index field, $facets. A NONE dimension still writes its doc values into that field but no StringField terms, so a document whose dimensions in the field are all NONE gives $facets index options NONE, while a document with any other dimension gives it DOCS. Since Lucene 9.0 (LUCENE-9334) a field must be indexed the same way in every document, so IndexWriter.addDocument / updateDocument throws IllegalArgumentException for the second kind of document: with the NONE-only document first for taxonomy, sorted-set and association facets alike, and in either order for taxonomy facets (one sorted-set order is accepted on 10.5.0 and later only because of a separate core bug). Once one kind is committed, a new IndexWriter keeps rejecting the other kind until the index is cleared with deleteAll() or rebuilt. Nothing is corrupted and no result is wrong, but the failure depends on the data, so it can first appear in production. The same layout works on 8.9.0 to 8.11.4, the releases that introduced NONE, and it is the layout of upstream's own NONE tests, which pass only because every test document also has a term-indexing dimension. The workaround, setIndexFieldName for the NONE dimensions, is not documented. Two fixes in FacetsConfig were validated: a fail-fast check with javadoc (recommended, no format change), and an indexed no-terms field that keeps the field's schema uniform (makes the layout work, but changes how such documents are indexed).

• Dimension and index field. A facet dimension (rating, author) has a DimConfig. Its indexFieldName is the Lucene field that receives the dimension's doc values (taxonomy ordinals, sorted-set labels or association bytes) and its drill-down terms. It defaults to DEFAULT_INDEX_FIELD_NAME = "$facets", the field that "holds the drill-downs and ords (as doc values)" (FacetsConfig.java:61-62, :126-130).
• Drill-down terms. StringField terms such as author and author\u001FBob that DrillDownQuery searches. DrillDownTermsIndexing picks which ones a dimension writes, per dimension (setDrillDownTermsIndexing(dimName, ...), :216-226); NONE means "Index no drill down terms" (:74-79). The default is ALL.
• Per-field schema. Since 9.0 (LUCENE-9334) every document must index a field with the same index options and doc-values type. Doc values alone mean index options NONE; a StringField means DOCS.

1. build() groups the facet fields by index field and calls the taxonomy, sorted-set or association path, which adds the doc values to indexFieldName (:412, :480-497, :532) and calls indexDrillDownTerms for each dimension (:400, :501, :530).
2. indexDrillDownTerms adds StringField(indexFieldName, ...) terms for every mode except NONE: if (dimConfig.drillDownTermsIndexing != DrillDownTermsIndexing.NONE) { (:419-460, :421).
3. So a document whose dimensions in $facets are all NONE has only doc values there (index options NONE); any other document has doc values plus terms (DOCS). Inside one document the core merges the two (IndexingChain.java:2278-2289), so a document with both dimensions is fine.
4. Across documents the core compares: assertSameSchema in a segment (IndexingChain.java:2345-2346, message at :2233-2234, :2258-2272) and verifySameIndexOptions across segments and after a commit (FieldInfos.java:511-515, FieldInfo.java:276-286). The second kind of document is rejected (from 10.5.0 a frozen-field-type cache can skip the in-segment check, IndexingChain.java:621-637, :1876-1896, which is why one sorted-set order is accepted; LFV-118). build() checks none of this; it does reject mixed association types in one field, with "use FacetsConfig to change the indexFieldName for each dimension" (:316-337).

Executed on main a9779ef3c193, assertions on and off (identical); rating is NONE, author the default, both in $facets. The 8.x column and the per-release results come from a version-independent main() that records only accept or reject; counts and drill-down were checked on main and 10.5.2:

Two IntAssociationFacetField dimensions sharing one field, one of them NONE, fail the same way with the NONE-only document first (the only order run). After the commit case, rating-only documents still go in and every author document is rejected; deleteAll() resets the schema. On 10.5.0 and later one order, a sorted-set author document and then a rating-only document in the same in-memory segment, is accepted, but only because of a separate core bug (LFV-118, a frozen-field-type validation cache); 9.0.0 to 10.4.0 reject it.

• Who. Applications that set DrillDownTermsIndexing.NONE on some, but not all, dimensions mapped to one index field, by default $facets, with taxonomy (FacetField), sorted-set (SortedSetDocValuesFacetField) or association facets (all three executed; for associations only IntAssociationFacetField, whose code path the float and bytes variants share). Ordinary sparse data triggers it: one document whose dimensions in the field are all NONE, and another with a term-indexing dimension there.
• Effect. addDocument / updateDocument throws a clean, non-aborting IllegalArgumentException and the document is not added as a live document. Nothing is corrupted and no search result is wrong. As with any non-aborting exception, IndexWriter still uses up a document ID for it and marks it deleted (DocumentsWriterPerThread.java:283-291); with taxonomy or association facets, build() has already added its labels to the taxonomy index (FacetsConfig.java:383, :517), where they stay unused, and deleteAll() on the main index does not remove them. Whichever kind of document is committed first decides which kind is rejected afterwards, until deleteAll() or a reindex.
• Default vs opt-in. NONE is opt-in. Once it is set, the failing layout is the default one, and it is the layout of upstream's own NONE tests.
• Scale. None needed: two documents, deterministic, assertions on and off.
• Not run. x86, concurrent writers, and a dedicated JUnit for a hierarchical NONE dimension (the bounded harness covered hierarchical configurations). All runs were on Linux aarch64.
• Workaround. setIndexFieldName for each NONE dimension. It is not free: counting then needs a separate Facets instance or SortedSetDocValuesReaderState per index field. The setIndexFieldName javadoc says the field name is "only used by the taxonomy based facet methods" (FacetsConfig.java:203-206), which is stale and hides the workaround from sorted-set users.
• How likely. Plausible, not common: one downstream code base found exposes NONE per field with the index field optional; nobody has reported it since 9.0.0 (2021).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected the author document to be indexed, but got: java.lang.IllegalArgumentException: Inconsistency of field data structures across documents for field [$facets] of doc [1]. index options: expected 'NONE', but it has 'DOCS'.
       org.apache.lucene.facet.TestDrillDownNoneSharedIndexField.testNoneOnlyDocThenDefaultDimDoc(TestDrillDownNoneSharedIndexField.java:42)
     Caused by: java.lang.IllegalArgumentException: Inconsistency of field data structures across documents for field [$facets] of doc [1]. index options: expected 'NONE', but it has 'DOCS'.
       org.apache.lucene.index.IndexingChain$FieldSchema.raiseNotSame(IndexingChain.java:2259)
       org.apache.lucene.index.IndexingChain$FieldSchema.assertSame(IndexingChain.java:2254)
       org.apache.lucene.index.IndexingChain$FieldSchema.assertSameSchema(IndexingChain.java:2346)
       org.apache.lucene.index.IndexingChain.initAndValidateFields(IndexingChain.java:695)
       org.apache.lucene.index.IndexingChain.processDocument(IndexingChain.java:641)
       [...]

• The contract. NONE: "Index no drill down terms" (FacetsConfig.java:74-79); the setter takes a dimension name (:216-226); every dimension defaults to $facets (:126-130). The demo's facet package docs: "FacetsConfig has a per-dimension config of the Category Terms field, e.g. you can choose to not index them at all" (package-info.java:234-238, the "Category Terms Field" section). Their "Multiple Category Lists" section says one field for all category-list (doc values) information "will suit most applications" (:355-357); that supports the shared default layout, but it is about the doc values, not the drill-down terms. No javadoc, CHANGES entry or MIGRATE note asks for a separate field.
• Upstream's tests. TestDrillDownQuery.testDrillDownTermsDifferentOptions puts NONE dimension a in $facets next to term-indexing dimensions b to e; TestSortedSetDocValuesFacets.testDrillDownOptions and testHierarchicalDrillDownOptions put NONE dimension c next to d to g. Every test document carries a term-indexing dimension, so they never build a NONE-only document. Two came with the feature (LUCENE-9385); testHierarchicalDrillDownOptions was added in 2022 by LUCENE-10250 (#509, first released in 9.1.0), after LUCENE-9334, with the same shared layout. They are still the only upstream tests that set NONE.
• History. LUCENE-9385 (#25) added NONE on 2021-03-27 (8.9.0); LUCENE-9334 (#11) made the core check exact on 2021-04-14 (9.0.0). Before it, the core silently upgraded a doc-values-only field to the indexed options. Neither PR discussion mentions the other.
• Precedent. LUCENE-9334 broke another facet layout, the taxonomy index's full-path field (LUCENE-9450, #10490); upstream fixed that in the facet module before 9.0.0.
• The module's convention. build() already rejects mixed association types in one field and tells the user to change the indexFieldName (:316-337). NONE has no such check.

The reviewers built the strongest case for "works as designed" (the refute-lens reviewer rated it strong, including its reading that applications must keep the dimensions of one field compatible, and still judged the bug real with the defect in FacetsConfig; the intended-lens reviewer rated it moderate; the devil's advocate found it does not hold): LUCENE-9334 deliberately fixes one schema per field (CHANGES: "A field across all documents within an index must be indexed with the same index options and data-structures"); in a 2021 dev@ thread a maintainer confirmed that documents with terms and doc values on a field require later documents to have "both terms and doc values on this field too, or nothing"; FacetsConfig is @lucene.experimental; setIndexFieldName exists; in LUCENE-7518 (2016) a maintainer wrote "One must use different field names" for taxonomy vs sorted-set doc values; and DEFAULT_INDEX_FIELD_NAME's javadoc says the field holds the drill-downs.

Why it fails: the core rule is not in dispute. The defect is that FacetsConfig turns a documented per-dimension option and its own default field into documents the core must reject, with no validation or caveat. LUCENE-7518 was a static conflict between two facet methods, hit by the first mixed document; here one method and one option give a conflict that depends on which dimensions each document carries. @lucene.experimental allows API changes, not an option that silently needs an undocumented layout. These points shape the fix, not whether it is a bug. Verdict: not intended, a defect.

Both options A and B were applied to a scratch worktree of main a9779ef3c193 and validated through gradle: the repro, their own or the hunt's tests, the whole lucene/facet module, the demo facet examples, and the lint, format and javadoc checks. A is recommended because it changes no index format and can be backported, though it is not behaviour-preserving in every case (see its trade-offs); B is the option that makes the shared layout work, at the cost of a compatibility break that was executed. C and D were not run.

+43/-2 in FacetsConfig.java. build() notes, per index field, whether this document indexes drill-down terms into it (any dimension there not NONE); a per-instance map keeps the first answer per field, the same best-effort pattern as the existing association-type check (assocDimTypes). A document with the other answer is rejected before any taxonomy ordinal is added. A NOTE on NONE states the rule and points to setIndexFieldName; the stale setIndexFieldName javadoc is corrected.

for (Map.Entry<String, Boolean> ent : hasDrillDownTerms.entrySet()) {
  Boolean previous = drillDownTermsByField.putIfAbsent(ent.getKey(), ent.getValue());
  if (previous != null && previous.booleanValue() != ent.getValue().booleanValue()) {
    throw new IllegalArgumentException(
        "indexed field \"" + ent.getKey() + "\" would get drill down terms in some documents"
            + " and none in others, because dimensions that use DrillDownTermsIndexing.NONE"
            + " share it with dimensions that index drill down terms; a field must be indexed"
            + " the same way in every document, so use FacetsConfig to change the"
            + " indexFieldName for the NONE dimensions");
  }
}

Trade-offs

• No index-format change; backportable to 10.x. Not behaviour-preserving in every case: it also rejects the sorted-set order "both dimensions, then a NONE-only document" in one in-memory segment, which stock 10.5.0+ and main accept only because of LFV-118 (the independent reproduction's test, now 5 of 8 failing), plus the shared-config cases below.
• It does not make the shared layout work: the error still comes when the second kind of document arrives, now from build(), before IndexWriter and the taxonomy see it, with a message that names the field and the remedy.
• Best effort: the map sees only documents this FacetsConfig built. After a restart against an index that already holds the other kind, the core's "cannot change field" error remains (the javadoc then explains it). One FacetsConfig shared by several indexes, or reused after deleteAll(), could reject a document that a particular index would accept.
• The map records a document's answer when build() runs, not when IndexWriter accepts it: a document that is built but never added, or that fails later (a null taxonomy writer, or an IndexWriter rejection for another reason), still fixes the field's answer, and with several index fields the earlier fields' answers are recorded before a later field throws.
• A stricter, deterministic check at configuration time ("NONE must not share a field with a term-indexing dimension") would break layouts that work today: upstream's own NONE tests, and indexes where every dimension in $facets is NONE.

What the checks validated

• Its test TestFacetsConfigDrillDownNoneCheck (5 tests: both orders rejected by build() with the new message; controls: every document has a term-indexing dimension, every dimension NONE, NONE in its own field) passes with assertions on and off.
• The repro, the finder's and the independent reproducer's tests now fail inside build() with the new message: 12 failures across the three files, none from IndexWriter.
• The whole lucene/facet module with the new test: 311 tests, 0 failed, 8 skipped. Demo facet examples: 19 tests, 0 failed. ecjLint, checkGoogleJavaFormat and renderJavadoc: BUILD SUCCESSFUL.

+66/-3 in FacetsConfig.java. When all of a document's dimensions in an index field are NONE and some dimension may index drill-down terms into that field (the default DimConfig's field, or a configured term-indexing dimension's field), build() adds new Field(indexFieldName, <empty TokenStream>, <StringField's index options, tokenized>). The field is then indexed (DOCS) in every document, and the NONE dimensions still add no terms. Same javadoc fixes.

Trade-offs

• Makes the documented default layout work, with counts and drill-down unchanged.
• Compatibility break, executed: an index written before the fix in which the shared field never had terms (for example every dimension in $facets is NONE) accepts more documents on stock, but after the fix its next NONE-only document is rejected with "cannot change field "$facets" from index options=NONE to inconsistent index options=DOCS". A new index with that layout works with the fix. So it needs a MIGRATE entry and suits a major release, not a 10.x backport.
• Adds an unusual empty indexed field; reviewers may prefer a fail-fast rule.
• Not measured: indexing cost, RAM, segment metadata or merge effects of the empty field. Its predicate looks at the configuration, not the index: it adds the field when the default DimConfig or any configured term-indexing dimension maps to that field, even if no such dimension ever appears in the index (the compatibility case above).

What the checks validated

• The repro (7), the finder's (5) and the independent reproducer's (8) tests: 20 of 20 pass, assertions on and off.
• The whole lucene/facet module with the three repro files: 326 tests, 0 failed, 8 skipped. Demo facet examples: 19 tests, 0 failed. Lint, format and javadoc: BUILD SUCCESSFUL.
• Compatibility probe (sorted-set facets, every dimension NONE, on disk): stock then stock accepted; fix then fix accepted; stock then fix rejected, as above.

Write a NONE dimension's doc values into a separate, derived field, so the shared field never has a doc-values-only occurrence.

Trade-offs

Not run. Changes where the counts of those dimensions are read (FastTaxonomyFacetCounts and DefaultSortedSetDocValuesReaderState read one field), so it needs search-side changes and changes the layout of existing indexes. Not recommended.

The javadoc part of the fail-fast option: a NOTE on NONE and the corrected setIndexFieldName javadoc.

Trade-offs

Cheapest; leaves the late, data-dependent core error as the only signal.

Method: semi-formal, bounded-exhaustive translation validation of the FacetsConfig option matrix, in a hunt unit on FacetsConfig and drill-down indexing (2026-10-27, main a9779ef3c193).

• Harness. For two dimensions with independently drawn options (drill-down mode, hierarchical, multi-valued, dimension count), taxonomy or sorted-set facets and three field layouts (shared $facets, shared custom field, separate fields), it indexed every multiset of one or two documents of up to two labels each, in four segment layouts (one or two segments, with or without a deleted document), with the stock jars, and compared accept/reject decisions, counts, drill-down and drill-sideways results with a reference model written from the javadoc. Mixed tier: 300 seeded configurations, 265,819,726 checks, assertions off.
• Signal. The class "build rejected a legal document" appeared in 41 of the 300 configurations, in no separate-field or same-mode one. Each of the 41 has one dimension with NONE and the other with a term-indexing mode in a shared field, and they are all such configurations the oracle checked: it counts a rejection only when dimension a is multi-valued, and 76 of the 300 configurations mix the two modes in a shared field.
• Witness and checks. The finder then wrote a public-API JUnit on the stock jars, and the claim went through an independent reproducer, reviews by a model from another family (OpenAI GPT-5.6, refute and intended lenses), an intended-behaviour devil's advocate, an adjudication and a credibility audit. Bounded scope only; no model checker or prover was involved.