FieldExistsQuery#count undercounts documents whose norms-free text field analyzes to no tokens but still has a doc value

FieldExistsQuery's Weight#count uses the number of documents with at least one indexed term as the number of documents with a doc value. For a tokenized field with norms omitted that shares its name with a doc-values field, a value that analyzes to no tokens (an empty, punctuation-only or stop-words-only text) gives a document that matches the query but is not counted. IndexSearcher#count then reports 1 for 2 hits, or 0 for 1 hit when no document of the segment has a term. Released since 9.1.0; not fixed upstream.

1. What the query matches. For a field without norms or vectors, scorerSupplier iterates the field's doc values (FieldExistsQuery.java:170-219, the doc-values switch at :188-209), so every document with a doc value is a hit. The class javadoc says the query matches documents that contain "a field that indexes norms or doc values" (:38-41).
2. What count() returns. count (:222-274) checks norms first and uses only the safe shortcut there (getDocCount(field) == maxDoc, :231-237). For a doc-values field without a skip index on a leaf without deletions (:247), the code says "No deletions: we can use points or terms doc count as a proxy for doc values" (:248) and an indexed field takes count = (terms == null ? 0 : terms.getDocCount()) (:252-255). Then count == 0 returns 0, count == maxDoc returns numDocs, and any other count of 0 or more is returned as exact ("The computed count can be trusted", :261-270).
3. Why the shortcut is wrong here. Terms#getDocCount is "the number of documents that have at least one term for this field" (Terms.java:117-121). A tokenized value that produces no tokens adds no term, but the document's doc value of the same name is there; the indexing chain handles a field that "exists in this document, but it did not have any indexed tokens" on purpose (IndexingChain.java:1930-1931). When no document of the leaf has a term, Terms is null and the result is 0.
4. Where the wrong number goes. IndexSearcher#count runs a TotalHitCountCollectorManager (IndexSearcher.java:491-529), whose leaf collector takes weight.count(context) whenever it is not -1 (TotalHitCountCollector.java:48). BooleanWeight#count composes clause counts: a FILTER clause's 0 makes the conjunction 0 (BooleanWeight.java:206-231), and under MUST_NOT the result numDocs - prohibitedCount comes out too high (:196).
5. What is sound. The points shortcut (every point field instance indexes a point), the skip-index branch (:244-246, an exact DocValuesSkipper#docCount), and the rewrite to MatchAllDocsQuery (:140-154, only when, on every leaf, the terms, points or skip-index doc count equals maxDoc). An untokenized value (StringField) always yields exactly one term, so the shortcut is exact for keyword fields, the shape it was written for.

Two documents, title = "" and title = "hello", each with a SortedDocValuesField("title", ...), norms omitted: 2 hits, count 1. When the only titled document of the segment is blank: 1 hit, count 0. The unbounded range SortedDocValuesField.newSlowRangeQuery("title", null, null, true, true) rewrites to the exists query: 2 hits, count 1. Stop-words-only values over 3 segments: 6 hits, count 3. #exists #id:1 on a segment where no document has a term: 1 hit, count 0. +*:* -exists over "", "hello" and a document without a title: 1 hit, count 2.

• Who is hit. A tokenized indexed field with omitNorms(true) and a doc-values field of the same name (SortedDocValuesField, SortedSetDocValuesField, NumericDocValuesField, ...) without a skip index, for example to sort or facet on the same name. Values that analyze to no tokens are ordinary data: empty strings, whitespace or punctuation only, stop words only with a stop filter. Default IndexWriterConfig, codec and IndexSearcher. The field shape is an opt-in choice.
• What is wrong. Counts only: IndexSearcher#count, TotalHitCountCollector(Manager), BooleanQuery counts built from the clause (too low under FILTER, too high under MUST_NOT), and unbounded doc-values range queries that rewrite to FieldExistsQuery. Hit lists, top-k totalHits and CheckIndex are fine.
• Scale. Per segment without deletions the count misses every zero-token document; a segment whose titled documents are all empty counts 0. Segments with deletions scan and are right.
• Not affected. Fields with norms (from 9.2.0), keyword fields such as StringField, point fields, and doc values with a skip index from 10.5.0.
• Workarounds today. Keep norms on the text field; or give the doc-values field a skip index (10.5.0 and later; that branch runs first and is exact); or use different names for the text field and the doc-values field (a doc-values-only field has no terms shortcut and is scanned; from the code, not run).
• Elsewhere. Elasticsearch hit the same mechanism in its own count shortcut (elastic/elasticsearch#89760: 3 total hits for 4 hits on an exists query with an empty string) and fixed it there in #90341.

The test sits in org.apache.lucene.search like Lucene's own tests but uses only public API. new IndexWriterConfig() brings StandardAnalyzer, which turns "" into no tokens; whitespace, punctuation-only or stop-words-only values behave the same. Single thread, deterministic: the two documents land in one segment and nothing is deleted. It compiles from 9.2.0 on (FieldExistsQuery and the org.apache.lucene.tests packages). For a release, compile and run the same file against that release's lucene-core, lucene-codecs and lucene-test-framework jars.

Line 47 (2 hits) passes; line 48 (count) fails. The same test is the suggested regression test; it passes with the recommended fix.

# gradle, main 5adff78bb168, seed DEADBEEF (codec Lucene104), assertions on
TestFieldExistsQueryCountNoTokens > testCountIncludesValueWithNoTokens FAILED
    java.lang.AssertionError: expected:<2> but was:<1>
        at org.junit.Assert.assertEquals(Assert.java:633)
        at org.apache.lucene.search.TestFieldExistsQueryCountNoTokens.testCountIncludesValueWithNoTokens(TestFieldExistsQueryCountNoTokens.java:48)

# JUnit runner, seed DEADBEEF: the same failure on main (-ea and -da), 10.6.0 (-ea and -da), 10.5.2, 9.12.3 and 9.2.0
# main, 20 iterations with random method seeds: 20 of 20 fail
# main with the recommended fix: passes (-ea and -da)

• Contract. Weight#count: "Counts the number of live documents that match a given Weight#parentQuery in a leaf"; overrides should "provide other accurate sub-linear implementations (that actually return the count)" (Weight.java:181-198). IndexSearcher#count: "Count how many documents match the given query" (IndexSearcher.java:491-494). Terms#getDocCount: "the number of documents that have at least one term for this field" (Terms.java:117-121).
• The query disagrees with itself. The match set is the query's own doc-values iterator (FieldExistsQuery.java:188-209), not an outside idea of what "exists" means.
• The twin was treated as a bug. In #477 (LUCENE-10263, 2021, eleven days after #445) jpountz pointed out that zero-token values get a norm "while they wouldn't count as part of Terms#getDocCount which only includes documents that have at least one term", and the norms shortcut was limited to docCount == maxDoc. The upstream test still says "docs that have a text field that analyzes to an empty token stream still have a recorded norm value but don't show up in Reader.getDocCount(field), so we can't use the shortcut for these fields" (TestFieldExistsQuery.java:577-580) and asserts -1 there. The doc-values branch from #445 was not revisited.
• History. #445's issue based the shortcut on per-field schema consistency (#10374, LUCENE-9334); the review suggested it "for keyword fields"; its test covers the terms shortcut only with StringField + SortedDocValuesField (and the points shortcut with LongPoint + NumericDocValuesField). Nothing in #445 discusses tokenized fields or values without tokens. In #16111 (2026) the author raised the empty-token caveat ("docs can have norms without postings when analyzer produces empty token stream") and concluded "the unified logic works for vectors/docValues/points/terms, but norms still only use the maxDoc shortcut".
• Legal input. IndexWriter accepts the field shape and the values, and the index passes CheckIndex.

The strongest case for "intended": the shortcut was a reviewed optimization for keyword fields; LUCENE-9334 makes a field use the same data structures in every document; in #477 a maintainer argued that zero-token values are an analysis-chain problem ("If you want tokens, index your data correctly"); and testDocValuesQueryMatchesCount (TestFieldExistsQuery.java:347) pins the O(1) count, so maintainers could document a limit instead. The works-as-designed reviewer rated this "moderate".

Why it fails: LUCENE-9334 checks which data structures a field uses per document, not that each document produces a term, and the indexing chain handles the zero-token case on purpose; no javadoc, CHANGES or MIGRATE entry limits the count to untokenized fields; the same argument did not keep the norms shortcut in #477; and in 2019 jpountz wrote that this optimization is unsafe unless the indexed and the doc-valued documents are the same (elastic/elasticsearch#37475). What is left is a fix trade-off, not an intent argument. Verdict: not intended, a defect.

Four options. A was built and validated on main 5adff78bb168; B was run only as a cause check; C and D were not built.

In the doc-values branch, use Terms#getDocCount only to show that every document of the leaf has the field, the rule the norms branch already follows since #477. Otherwise count stays -1 and IndexSearcher#count collects the hits. The points and skip-index branches are unchanged.

} else if (fieldInfo.getIndexOptions() != IndexOptions.NONE) {
  // Terms#getDocCount only counts documents that have at least one term, but a document
  // can have a doc value and no term (a tokenized value that produced no tokens). So it
  // can only show that every document has the field, like the norms shortcut above.
  Terms terms = reader.terms(field);
  if (terms != null && terms.getDocCount() == reader.maxDoc()) {
    count = reader.maxDoc();
  }
}

Trade-offs

• On deletion-free segments where only some documents have a keyword field (StringField + same-name doc values), the count becomes a scan of the doc-values iterator instead of O(1). Segments where every document has the field keep O(1); users can get O(1) back with a doc-values skip index (10.5.0 on). No benchmark was run.
• One assertion of an upstream test changes, so this is the maintainers' call.

What the checks validated

• Gradle in a clean scratch worktree of 5adff78bb168, OpenJDK 25.0.3, seed DEADBEEF: stock, the repro fails; with the fix it passes with assertions on and off.
• With the source change alone, TestFieldExistsQuery has 1 failure in 30 tests: testDocValuesQueryMatchesCount, whose assertSameCount(..., "string", ...) (TestFieldExistsQuery.java:374) expects the O(1) Weight#count for a StringField that only some documents have. The failure reads expected:<-1> but was:<44> because the helper passes its arguments in reverse (:888-890).
• With that assertion changed to check searcher.count only, the whole org.apache.lucene.search package passes: 216 test classes, 2,351 tests, 0 failures, 49 skipped.
• :lucene:core:check -x test (format, ecj lint, forbidden APIs) passes.
• An equivalent earlier sketch passed the original 4-test repro (4/4) and an independent 7-test repro (7/7) on ea57360912ed.

Delete the three-line branch. As a cause check this made all four tests of the original repro pass on main and on 9.1.0.

Trade-offs

Also drops the safe == maxDoc case: strictly more de-optimization than A.

FieldInfo does not record whether a field is tokenized, so a read-time fix cannot tell StringField from a norms-free TextField. Recording per segment that every document with the field produced a term, or the number of documents with a doc value, would keep the shortcut exactly where it is sound.

Trade-offs

Needs a format or API change; not built.

Would keep StringField (indexed with DOCS) on the O(1) path.

Trade-offs

Not sound: a tokenized field can also be indexed with DOCS and produce no tokens (from the code, not run).

Method: semi-formal. The hypothesis came from reading FieldExistsQuery#count while building an Alloy 6.2 model of its shortcuts against the query's match set (the scorer's iterator).

• check countSound (4 documents, 5-bit integers) is satisfiable: the counterexample's field is indexed, tokenized, has doc values and no norms.
• check countSoundWithoutTokenizedNoNorms (5 documents) is unsatisfiable: with that shape excluded, every shortcut is exact in scope.
• Vacuity runs show all three exact branches (terms, points, norms) are reachable.
• An end-to-end JUnit test on the real jars then reproduced it, and a differential check that compares IndexSearcher#count with the collected hits on random indexes rediscovers it.