The classic Highlighter throws IllegalStateException for FieldExistsQuery when it highlights from term vectors

The classic Highlighter with a QueryScorer finds the words to mark by rewriting the query's clauses against a tiny one-field index of the document being highlighted. When the token stream comes from term vectors with positions and offsets, that index is a TermVectorLeafReader, whose only field (shadowed_field) has no norms, doc values or vectors, and since #15678 (10.5.0) its getFieldInfos() returns that field for every name. #15678 also removed the FieldExistsQuery skip that #12088 (9.5.0) had added. So a FieldExistsQuery on any field, or a query that rewrites to one such as LongField.newRangeQuery(f, Long.MIN_VALUE, Long.MAX_VALUE), reaches FieldExistsQuery.rewrite, which throws IllegalStateException naming shadowed_field. The highlight call fails for every hit; 10.4.0 returns the quick brown <B>fox</B>. Search results, re-analysed text and offsets-only term vectors are not affected. Restoring the skip fixes it and passes all 732 tests of the highlighter module with the repro tests added.

• Classic Highlighter: org.apache.lucene.search.highlight.Highlighter with a QueryScorer. The scorer asks WeightedSpanTermExtractor which terms of the query occur in the token stream of the text.
• Token stream from term vectors: instead of re-analysing the stored text, TokenSources can rebuild the tokens from the document's term vectors (getTermVectorTokenStreamOrNull, or getTokenStream(field, tvFields, text, analyzer, max), which prefers term vectors when they have offsets). This is the usual speed-up for this highlighter.
• Internal reader: for queries it has no special case for (apart from spatial queries and multi-term queries it does not expand), the extractor calls query.rewrite(...) on a one-document reader made from the token stream, then extracts terms from the result. With term vectors that have positions and offsets the reader is a TermVectorLeafReader; otherwise the text is indexed into a MemoryIndex. A DelegatingLeafReader wraps it and answers every field name with its one field, shadowed_field.
• FieldExistsQuery: matches documents that have a value in a field, using norms, doc values or vectors. Its rewrite throws IllegalStateException for a field that exists but indexes none of them.

// WeightedSpanTermExtractor.extract
} else if (query instanceof MatchAllDocsQuery) {            // :236
  // nothing
                                                          // 9.5.0-10.4.0 had here:
                                                          //   } else if (query instanceof FieldExistsQuery) { // nothing
} else if (query instanceof FunctionScoreQuery) {           // :238
  ...
} else {
  final IndexReader reader = getLeafContext().reader();     // :247  the internal one-field reader
  rewritten = query.rewrite(new IndexSearcher(reader));     // :254  FieldExistsQuery.rewrite runs here

// getLeafContext(), :397-399: term vectors with positions AND offsets
internalReader = new TermVectorLeafReader(DelegatingLeafReader.FIELD_NAME, termVectorTerms);

// DelegatingLeafReader.getFieldInfos(), :442-451 (since #15678): every name gets the one FieldInfo
public FieldInfo fieldInfo(String fieldName) { return super.fieldInfo(DelegatingLeafReader.FIELD_NAME); }

// TermVectorLeafReader.java:93-112: omitNorms = true, DocValuesType.NONE, vector dimension 0
// FieldExistsQuery.java:156: no norms, no vectors, no doc values ->
throw new IllegalStateException(buildErrorMsg(fieldInfo));

1. A BooleanQuery is walked clause by clause (prohibited clauses skipped), so MUST, FILTER and SHOULD clauses all reach extract.
2. body:fox gives the term fox. FieldExistsQuery(price) has no branch since 10.5.0, so it is rewritten on the internal reader.
3. Asked for price, the internal reader answers with shadowed_field, which on the term-vector path has no norms, doc values or vectors. The real price has numeric doc values.
4. FieldExistsQuery.rewrite throws; the exception leaves Highlighter.getBestFragment.
5. On the MemoryIndex path the same FieldInfo has norms, docCount 1 equals maxDoc 1, and the rewrite gives MatchAllDocsQuery, which the extractor ignores (traced).

The full-range LongField query gets there one step later: IndexSortSortedNumericDocValuesRangeQuery rewrites [Long.MIN_VALUE, Long.MAX_VALUE] to new FieldExistsQuery(field) (IndexSortSortedNumericDocValuesRangeQuery.java:131-133), which is extracted again (:259).

• Who. Users of the classic Highlighter with a QueryScorer who take token streams from term vectors with positions and offsets: TokenSources.getTermVectorTokenStreamOrNull, or TokenSources.getTokenStream(field, tvFields, text, analyzer, max) when the field has such vectors. Neither method is deprecated; TokenSources recommends positions on the term vector, and the package javadoc presents term vectors as the faster way to highlight.
• Trigger. Any clause of the highlighted query that is, or rewrites to, a FieldExistsQuery, on any field: exists filters on doc-values, norms, vector, keyword or numeric fields and on the highlighted field itself; full-range numeric and unbounded sorted or sorted-set doc-values ranges, and the full-range LongField range; an IndexOrDocValuesQuery whose index side is an exists query (the extractor reads only that side); the same under ConstantScore, Boost, FunctionScore, DisjunctionMax or nested Boolean queries. MUST_NOT clauses are skipped by the extractor and do not trigger it.
• Scale. The finder's matrix ran 7 document shapes x 3 text queries x about 130 query atoms x 5 occurrences (13,884 searchable queries) and highlighted every hit in 26 modes: 533,104 checks on main. 36 atoms throw this exception, all in the three QueryScorer term-vector modes and none in the UnifiedHighlighter or FastVectorHighlighter modes. (The matrix also records other, unrelated exceptions in some modes; they are not part of this report.) #16582 (10.6) makes ranges that cover every value of a field rewrite to FieldExistsQuery on the real index, but not on the highlighter's internal reader, which has no doc-values skipper: executed on main, such ranges still highlight from term vectors, and only the full Long.MIN_VALUE..Long.MAX_VALUE range throws, as on 10.5.2.
• Effect. One unchecked IllegalStateException per highlight call, so the request gets no highlights or an error. One document is enough, and no special settings are needed: it fails with the test framework's randomized directory and config (the repro) and with a plain IndexWriterConfig, the default codec and a plain IndexSearcher (the diagram probe); assertions on and off. Nothing is written and search results are correct.
• Opt-in. Term vectors with positions and offsets are an index-time choice, and the classic Highlighter is the older highlighter (#16820, an open proposal, would deprecate it in 11.0; nothing is deprecated yet). No matching public report was found in the four months since 10.5.0.
• Workaround. Highlight with a query that has no exists or full-range clauses, or override the protected QueryScorer.newTermExtractor to return a WeightedSpanTermExtractor whose protected isQueryUnsupported also returns true for FieldExistsQuery. Both give the quick brown <B>fox</B> on main from a package outside Lucene, with assertions on and off.

TestHighlightFieldExistsTermVectors (package org.apache.lucene.search.highlight, public API only). One document: body stored with term vectors with positions and offsets, price as a NumericDocValuesField, size as a LongField. Each test first asserts that the query finds the document, then highlights it from its term vectors.

• testTermVectorTokenStream: +body:fox #FieldExistsQuery(price); fails.
• testTermVectorTokenStreamOpenEndedLongRange: the full-range LongField.newRangeQuery("size", Long.MIN_VALUE, Long.MAX_VALUE) as the filter; fails (the method name says open-ended).
• testReanalysedTokenStream: the first query over re-analysed text; passes (control).

Two more tests are attached: the independent reproducer's (exists on a doc-values field, exists on the highlighted field, the full-range LongField range, two controls; 3 of 5 fail) and a variant test (exists on the highlighted field as a MUST clause, QueryScorer without a field, TokenSources.getTokenStream, offsets-only control; 1 of 2 fails). All three pass with the recommended fix.

In its own package verify.v1; it also runs in org.apache.lucene.search.highlight after changing only the package line. 3 of 5 fail on main; both controls pass.

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package verify.v1;

import org.apache.lucene.analysis.Analyzer;
import org.apache.lucene.analysis.TokenStream;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.document.FieldType;
import org.apache.lucene.document.LongField;
import org.apache.lucene.document.NumericDocValuesField;
import org.apache.lucene.document.TextField;
import org.apache.lucene.index.DirectoryReader;
import org.apache.lucene.index.Fields;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.index.Term;
import org.apache.lucene.search.BooleanClause;
import org.apache.lucene.search.BooleanQuery;
import org.apache.lucene.search.FieldExistsQuery;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.Query;
import org.apache.lucene.search.TermQuery;
import org.apache.lucene.search.highlight.Highlighter;
import org.apache.lucene.search.highlight.QueryScorer;
import org.apache.lucene.search.highlight.TokenSources;
import org.apache.lucene.store.Directory;
import org.apache.lucene.tests.analysis.MockAnalyzer;
import org.apache.lucene.tests.analysis.MockTokenizer;
import org.apache.lucene.tests.util.LuceneTestCase;

/**
 * Independent check (public API only, own package): the classic Highlighter, fed a token stream
 * built from term vectors with positions and offsets, should highlight the matching term of a
 * query that also contains a FieldExistsQuery (or a query that rewrites to one), as it does for a
 * re-analysed token stream.
 */
public class TestFieldExistsTermVectorHighlight extends LuceneTestCase {

  private static final String TEXT = "the quick brown fox";
  private static final String EXPECTED = "the quick brown <B>fox</B>";

  private Directory dir;
  private DirectoryReader reader;
  private Analyzer analyzer;
  private int docId;

  @Override
  public void setUp() throws Exception {
    super.setUp();
    analyzer = new MockAnalyzer(random(), MockTokenizer.WHITESPACE, true);
    dir = newDirectory();
    FieldType tvType = new FieldType(TextField.TYPE_STORED);
    tvType.setStoreTermVectors(true);
    tvType.setStoreTermVectorPositions(true);
    tvType.setStoreTermVectorOffsets(true);
    tvType.freeze();
    try (IndexWriter w = new IndexWriter(dir, new IndexWriterConfig(analyzer))) {
      Document doc = new Document();
      doc.add(new Field("body", TEXT, tvType));
      doc.add(new NumericDocValuesField("dv", 7L));
      doc.add(new LongField("num", 42L, Field.Store.NO));
      w.addDocument(doc);
    }
    reader = DirectoryReader.open(dir);
    docId = new IndexSearcher(reader).search(new TermQuery(new Term("body", "fox")), 1).scoreDocs[0]
        .doc;
  }

  @Override
  public void tearDown() throws Exception {
    reader.close();
    dir.close();
    analyzer.close();
    super.tearDown();
  }

  private static Query foxAnd(Query filter) {
    return new BooleanQuery.Builder()
        .add(new TermQuery(new Term("body", "fox")), BooleanClause.Occur.MUST)
        .add(filter, BooleanClause.Occur.FILTER)
        .build();
  }

  private TokenStream termVectorStream() throws Exception {
    Fields tv = reader.termVectors().get(docId);
    TokenStream ts = TokenSources.getTermVectorTokenStreamOrNull("body", tv, -1);
    assertNotNull("term vectors with offsets expected", ts);
    return ts;
  }

  private void assertHighlights(Query query, TokenStream ts) throws Exception {
    // precondition: the document matches the query on the real index
    assertEquals(1, new IndexSearcher(reader).count(query));
    Highlighter highlighter = new Highlighter(new QueryScorer(query, "body"));
    String fragment;
    try {
      fragment = highlighter.getBestFragment(ts, TEXT);
    } catch (RuntimeException e) {
      throw new AssertionError("highlighting threw for query " + query + ": " + e, e);
    }
    assertEquals(EXPECTED, fragment);
  }

  /** Bug claim: FieldExistsQuery on a doc-values field, term-vector token stream. */
  public void testTermVectorStreamFieldExistsOnDocValuesField() throws Exception {
    assertHighlights(foxAnd(new FieldExistsQuery("dv")), termVectorStream());
  }

  /** Bug claim: FieldExistsQuery on the highlighted field itself (it has norms). */
  public void testTermVectorStreamFieldExistsOnHighlightedField() throws Exception {
    assertHighlights(foxAnd(new FieldExistsQuery("body")), termVectorStream());
  }

  /** Bug claim: an open-ended long range, which rewrites to a FieldExistsQuery. */
  public void testTermVectorStreamOpenEndedLongRange() throws Exception {
    assertHighlights(
        foxAnd(LongField.newRangeQuery("num", Long.MIN_VALUE, Long.MAX_VALUE)),
        termVectorStream());
  }

  /** Control: same FieldExistsQuery query, re-analysed token stream. */
  public void testControlReanalysedStreamFieldExists() throws Exception {
    assertHighlights(
        foxAnd(new FieldExistsQuery("dv")), analyzer.tokenStream("body", TEXT));
  }

  /** Control: term-vector token stream, query without FieldExistsQuery. */
  public void testControlTermVectorStreamPlainTerm() throws Exception {
    assertHighlights(new TermQuery(new Term("body", "fox")), termVectorStream());
  }
}

D=$LUCENE/lucene/highlighter/src/test/org/apache/lucene/search/highlight
sed 's/^package verify\.v1;/package org.apache.lucene.search.highlight;/' TestFieldExistsTermVectorHighlight.java > $D/TestFieldExistsTermVectorHighlight.java
cd $LUCENE
./gradlew :lucene:highlighter:test --tests org.apache.lucene.search.highlight.TestFieldExistsTermVectorHighlight -Ptests.seed=DEADBEEF

Exists on the highlighted field as a MUST clause through TokenSources.getTokenStream; 1 of 2 fails, the offsets-only control passes.

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package org.apache.lucene.search.highlight;

import org.apache.lucene.analysis.Analyzer;
import org.apache.lucene.analysis.TokenStream;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.document.FieldType;
import org.apache.lucene.document.TextField;
import org.apache.lucene.index.DirectoryReader;
import org.apache.lucene.index.Fields;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.Term;
import org.apache.lucene.search.BooleanClause.Occur;
import org.apache.lucene.search.BooleanQuery;
import org.apache.lucene.search.FieldExistsQuery;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.Query;
import org.apache.lucene.search.TermQuery;
import org.apache.lucene.search.TopDocs;
import org.apache.lucene.store.Directory;
import org.apache.lucene.tests.analysis.MockAnalyzer;
import org.apache.lucene.tests.util.LuceneTestCase;

/**
 * Variants: the FieldExistsQuery targets the highlighted field itself (which indexes norms), the
 * clause is MUST, the token stream comes from the non-deprecated TokenSources.getTokenStream
 * convenience (term vectors first, analysis as fallback), and QueryScorer has no field restriction.
 */
public class TestFieldExistsTermVectorVariants extends LuceneTestCase {

  private static final String TEXT = "the quick brown fox";

  private static String highlight(boolean positions) throws Exception {
    Analyzer analyzer = new MockAnalyzer(random());
    try (Directory dir = newDirectory()) {
      FieldType tv = new FieldType(TextField.TYPE_STORED);
      tv.setStoreTermVectors(true);
      tv.setStoreTermVectorPositions(positions);
      tv.setStoreTermVectorOffsets(true);
      tv.freeze();
      try (IndexWriter w = new IndexWriter(dir, newIndexWriterConfig(analyzer))) {
        Document d = new Document();
        d.add(new Field("body", TEXT, tv));
        w.addDocument(d);
      }
      try (DirectoryReader reader = DirectoryReader.open(dir)) {
        Query q =
            new BooleanQuery.Builder()
                .add(new TermQuery(new Term("body", "fox")), Occur.MUST)
                .add(new FieldExistsQuery("body"), Occur.MUST)
                .build();
        TopDocs td = new IndexSearcher(reader).search(q, 10);
        assertEquals(1, td.scoreDocs.length);
        int doc = td.scoreDocs[0].doc;
        Fields tvFields = reader.termVectors().get(doc);
        String text = reader.storedFields().document(doc).get("body");
        TokenStream ts = TokenSources.getTokenStream("body", tvFields, text, analyzer, -1);
        Highlighter highlighter = new Highlighter(new QueryScorer(q));
        return highlighter.getBestFragment(ts, text);
      }
    }
  }

  /** Term vectors with positions and offsets: the TermVectorLeafReader path. */
  public void testTermVectorsWithPositionsAndOffsets() throws Exception {
    assertEquals("the quick brown <B>fox</B>", highlight(true));
  }

  /** Control: offsets without positions, so the extractor re-indexes into a MemoryIndex. */
  public void testTermVectorsWithOffsetsOnly() throws Exception {
    assertEquals("the quick brown <B>fox</B>", highlight(false));
  }
}

cp TestFieldExistsTermVectorVariants.java $LUCENE/lucene/highlighter/src/test/org/apache/lucene/search/highlight/
cd $LUCENE
./gradlew :lucene:highlighter:test --tests org.apache.lucene.search.highlight.TestFieldExistsTermVectorVariants -Ptests.seed=DEADBEEF

cp TestHighlightFieldExistsTermVectors.java $LUCENE/lucene/highlighter/src/test/org/apache/lucene/search/highlight/
cd $LUCENE
./gradlew :lucene:highlighter:test --tests org.apache.lucene.search.highlight.TestHighlightFieldExistsTermVectors -Ptests.seed=DEADBEEF
# assertions off (production-like): add -Ptests.asserts=false

# gradle, main a9779ef3c193, assertions on, seed DEADBEEF (the same 2 of 3 with -Ptests.asserts=false); abridged and reordered: gradle prints the range failure first, with module prefixes, SeedInfo and test frames
TestHighlightFieldExistsTermVectors > testTermVectorTokenStream FAILED
    java.lang.IllegalStateException: FieldExistsQuery requires that the field indexes doc values, norms or vectors, but field 'shadowed_field' exists and indexes neither of these data structures
        at org.apache.lucene.search.FieldExistsQuery.rewrite(FieldExistsQuery.java:156)
        at org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:254)
        at org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:136)
        at org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:520)
        at org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:209)
        at org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:178)
        at org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:175)
        at org.apache.lucene.search.highlight.Highlighter.getBestFragments(Highlighter.java:140)
        at org.apache.lucene.search.highlight.Highlighter.getBestFragment(Highlighter.java:98)

TestHighlightFieldExistsTermVectors > testTermVectorTokenStreamOpenEndedLongRange FAILED
    java.lang.IllegalStateException: (same message)
        at org.apache.lucene.search.FieldExistsQuery.rewrite(FieldExistsQuery.java:156)
        at org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:254)
        at org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:259)
        at org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:136)
        ...

testReanalysedTokenStream: passed   (from the gradle test results)

# with the attached two tests: 10 tests, 6 failed (assertions on and off); every control passes
# 10.5.0 / 10.5.2: the same exception at FieldExistsQuery.java:155 <- WeightedSpanTermExtractor.java:255

• #12088 (9.5.0), commit fc7b937affe "Don't throw UOE when highlighting FieldExistsQuery": "This commit makes WeightedSpanTermExtractor aware of FieldExistsQuery, so that it can just ignore it and avoid throwing an exception." Its test TestHighlighter.testFieldExistsQuery still exists.
• #15678 (10.5.0), commit a8b3d1a9481d: "This commit adds a more useful implementation, meaning that queries calling getFieldInfos() during rewrite will not throw exceptions during highlighting." Its PR body: the new FieldInfos "ensures that any current or future query types that rely on index schema metadata during rewrite can be handled gracefully by the highlighter". The issue it fixed, #15668, asked to end the "whack-a-mole" of query optimizations that break the highlighter.
• Why the removal slipped through (our reading; the thread does not discuss term vectors). On #15678 the reviewer asked "maybe let's remove the special-casing for FieldExistsQuery as well?" and the author did. #15678's two tests and testFieldExistsQuery take the MemoryIndex path, where the internal field has norms: testSortedNumericDocValuesRangeQuery and testFieldExistsQuery use a CannedTokenStream, and TestHighlightCustomQuery.testHighlightQueryRequiringMetadata re-analyses the text with a MockAnalyzer. No upstream test highlights a FieldExistsQuery from term vectors.
• Correct usage. Public, non-deprecated APIs only; the document matches the query on the real index; the error names shadowed_field, a field the user never created; it fires even for an exists clause on the highlighted field, which has norms.
• The class javadoc leaves queries it does not support to subclasses: they may override extract and extractUnknownQuery "to support additional, by default unsupported queries", and the default extractUnknownQuery, reached when a rewrite returns the query unchanged, does nothing. It does not promise that a rewrite cannot throw, so it is weaker support than the two commit messages.
• Stability markers, both ways. Highlighter, QueryScorer, WeightedSpanTermExtractor and the two TokenSources methods are public and not deprecated; TermVectorLeafReader and the FieldInfo constructor are @lucene.experimental, and TokenStreamFromTermVector is @lucene.internal.

Strongest case (a reviewer from another model family, OpenAI GPT-5.6, and an intended-behaviour checker): the classic Highlighter is a lexical term extractor that never promised to handle every query; FieldExistsQuery has no terms to highlight, and its exception is a deliberate fail-fast for a field without norms, doc values or vectors; the term-vector reader is a deliberately partial @lucene.experimental view; #15678 removed the special case on purpose; the #12088 CHANGES line names only UnsupportedOperationException; and #16820 proposes deprecating the classic Highlighter.

Why it fails: the fail-fast fires on the highlighter's own description of the schema, not on the caller's field (the real price has doc values); both commit messages state a no-exception intent for exactly this kind of rewrite; the removal was justified only for the path its tests covered; 10.4.0 behaves differently with the same code and query; and a proposed deprecation is not a decision. Both reviewers rated the argument weak.

Fix A was built and run in a scratch worktree at a9779ef3c193 with the whole :lucene:highlighter:test task and the format and lint tasks; fix B with the same tests (assertions on). Fix C was run by the independent reproducer; the PR #16816 change was run as a shadow class; the workarounds were run on main from a package outside Lucene.

Put back the branch #12088 added and #15678 removed, right after the MatchAllDocsQuery branch.

+import org.apache.lucene.search.FieldExistsQuery;
 ...
     } else if (query instanceof MatchAllDocsQuery) {
       // nothing
+    } else if (query instanceof FieldExistsQuery) {
+      // nothing: it has no terms to highlight, and its rewrite reads FieldInfos that the internal
+      // reader cannot report faithfully (term vectors have no norms or doc values)
     } else if (query instanceof FunctionScoreQuery) {

Trade-offs

• Restores 10.4.0's handling of FieldExistsQuery (main keeps #15678's working getFieldInfos()); one instanceof check per extracted query (not benchmarked).
• Narrow: other rewrites can still read the misleading FieldInfo. In the matrix it fixes 32 of the 36 throwing atoms; the other 4 are kNN queries with a filter or seed whose own rewrite runs a search on the internal reader, a related problem with a different fix that is better filed and cross-linked separately.

What the checks validated

• Gradle on a9779ef3c193, seed DEADBEEF: the 3 + 5 + 2 repro tests pass, and :lucene:highlighter:test passes with assertions on and off: 37 classes, 732 tests, 0 failures (722 upstream tests, e.g. TestHighlighter 59, TestTokenSources 8, TestUnifiedHighlighterTermVec 6).
• checkGoogleJavaFormat, forbiddenApisMain and ecjLintMain pass.
• Earlier, with the standalone runner: the finder's, the independent reproducer's and the second reviewer's tests pass, and 81 upstream classic-highlighter tests.

Pass omitNorms = false in TermVectorLeafReader's FieldInfo; FieldExistsQuery.rewrite then takes its norms branch and no longer throws.

Trade-offs

• The reader then claims norms while getNormValues still returns null.
• TermVectorLeafReader is a public @lucene.experimental class, also used by the UnifiedHighlighter's term-vector strategy.
• The exists clause is still evaluated on the shadowed field, not the named one.

What the checks validated

• Gradle on a9779ef3c193, assertions on: all 732 tests pass. Not run with assertions off.

DelegatingLeafReader.getFieldInfos().fieldInfo(name) returns null for fields other than the highlighted one (the fix sketch of the related search-executing-rewrite report).

Trade-offs

Leaves FieldExistsQuery on the highlighted field itself failing: run by the independent reproducer on main, 1 of 5 still fails. Useful only together with A.

#16816 ("Rewrite special case AutomatonQueries", open; labelled Stale by the stale bot on 2026-10-28, last human activity 2026-10-13, merge conflicts) makes FieldExistsQuery.rewrite accept fields that index only terms.

Trade-offs

With its FieldExistsQuery.java as a shadow class the repro passes 3 of 3, so merging it would hide this bug on main by accident; 10.5.x and 10.6.x stay affected.

Until a fix ships: highlight with a query that has no exists or full-range clauses, or override the protected QueryScorer.newTermExtractor to return a WeightedSpanTermExtractor whose protected isQueryUnsupported also returns true for FieldExistsQuery.

new QueryScorer(query, "body") {
  @Override
  protected WeightedSpanTermExtractor newTermExtractor(String defaultField) {
    return new WeightedSpanTermExtractor(defaultField) {
      @Override
      protected boolean isQueryUnsupported(Class<? extends Query> clazz) {
        return clazz == FieldExistsQuery.class || super.isQueryUnsupported(clazz);
      }
    };
  }
}

Trade-offs

Application code only; it does not fix Lucene, and every application on the term-vector path needs it.

What the checks validated

• Main a9779ef3c193, a test in a package outside Lucene, assertions on and off: both workarounds give the quick brown <B>fox</B> for the exists filter (and the extractor override for the full-range LongField filter); a stock QueryScorer throws in the same test. 4 of 4 pass.

• First signal (non-formal). The hunt's planning analysis read the DelegatingLeafReader remap that #15678 added, prompted by downstream reports of its predecessor UnsupportedOperationException, and wrote a small probe: a term-vector field plus a FieldExistsQuery filter. It threw on main and was bisected to 10.5.0 by release.
• Reduction. A hunt unit on the classic and fast-vector highlighters turned it into the public-API JUnit of the repro and ran it on main and the release jars.
• Widening (bounded-exhaustive checking of the stock jars). A trigger model enumerated 11 token-stream cells (term-vector flag combinations, non-empty and empty text, the analysis path with and without caching and payloads) and checked "highlighting throws iff the internal reader is a TermVectorLeafReader" within them (holds, 0 violations). A matrix ran 7 document shapes x 3 text queries x about 130 query atoms x 5 occurrences and highlighted every hit in 26 modes: 533,104 checks on main (455,052, with a smaller atom set, on 10.5.2 and 10.4.0), grouped by throwing frame: 36 atoms throw this exception on main, only in the classic Highlighter's term-vector modes; 10.4.0 has none.