MemoryIndex ignores TermFrequencyAttribute, so FeatureField values and custom term frequencies are indexed as frequency 1

FeatureField stores its value as a term frequency, and any TokenStream can set a custom frequency through TermFrequencyAttribute (for example DelimitedTermFrequencyTokenFilter on a|7, or a DOCS_AND_CUSTOM_FREQS field since 10.5.0). For DOCS_AND_FREQS and DOCS_AND_CUSTOM_FREQS fields IndexWriter indexes that frequency; MemoryIndex.storeTerms never reads the attribute and adds 1 per token to the postings frequency, sumTotalTermFreq and the field length. A document put into a MemoryIndex (fromDocument, addField(IndexableField, Analyzer), or Lucene Monitor matching one document) therefore counts each such token as frequency 1: FeatureField("features", "pagerank", 10f) is frequency 33344 in an IndexWriter index and 1 in a MemoryIndex, which decodes to 4.5918E-41. FeatureField.newLinearQuery scores 4.5918E-41 instead of 10.0, and newSaturationQuery with the pivot 10 scores exactly 0.0 instead of 0.5 although the document matches; 0.0 is what MemoryIndex.search documents as "no match". Since 8.6.0, Monitor scores the same document 10.0 in a two-document batch and 4.5918E-41 alone, and drops a stored saturation query with the pivot 10 for the single document. For the tested queries the set of matching documents (IndexSearcher.count) is unchanged; only score-based match tests lose the match. Every release checked from 7.0.0 to 10.6.0 (10.6.0 by reading) is affected; the validated fix reads the attribute where IndexWriter honours it and rejects overflow as IndexWriter does.

• TermFrequencyAttribute: a token attribute that "Sets the custom term frequency of a term within one document"; the field must be indexed with DOCS_AND_FREQS (TermFrequencyAttribute.java:24-26 on main a9779ef3c193). DelimitedTermFrequencyTokenFilter sets it from text like a|7; since 10.5.0 a DOCS_AND_CUSTOM_FREQS field treats it as a per-term score, adding 1 per term to the field length.
• FeatureField: a rank-feature field (pagerank, learned-sparse term weights). "Feature values are internally encoded as term frequencies" (FeatureField.java:47): the value becomes Float.floatToIntBits(value) >>> 15 (:185-186; 10f gives 33344), set through the attribute (:222) on a DOCS_AND_FREQS field (:114). Its queries decode the frequency back with Float.intBitsToFloat(freq << 15) (:240-249).
• MemoryIndex: a single-document index held in memory, "a replacement/substitute for RAM-resident Directory implementations" (MemoryIndex.java:106). Lucene Monitor uses one for a one-document batch and an IndexWriter-built reader for larger batches (DocumentBatch.java:44-62, 105-113).

1. IndexWriter uses the attribute, for DOCS_AND_FREQS and DOCS_AND_CUSTOM_FREQS fields. FreqProxTermsWriterPerField.getTermFreq (FreqProxTermsWriterPerField.java:208-219) takes it as the postings frequency, and IndexingChain adds it to the field length (1 per term for DOCS_AND_CUSTOM_FREQS) with Math.addExact (IndexingChain.java:2053-2064). A DOCS field keeps no frequencies (distinct terms read as 1 on both paths), and a positional field throws "cannot index positions while using custom TermFrequencyAttribute" (:210-216).
2. MemoryIndex does not. storeTerms (MemoryIndex.java:991-1053) asks the stream only for the term, position-increment, offset and payload attributes (:1002-1007) and per token does info.numTokens++ (:1012), info.sliceArray.freq[ord]++ (:1025) and info.sumTotalTermFreq++ (:1027). The norm is computed from numTokens (:1380-1393). TermFrequencyAttribute appears nowhere in the file.
3. Frequency 1 decodes to almost nothing. Float.intBitsToFloat(1 << 15) is 4.5918E-41, a subnormal float below the smallest value FeatureField accepts (Float.MIN_NORMAL, FeatureField.java:162-171). The linear function returns it. The other functions depend on their parameters (executed): saturation weight * (1 - pivot / (f + pivot)) (:450) with the pivot 10 rounds 10f / (10f + 4.5918E-41f) to 1.0 and returns exactly 0.0, and so does sigmoid with the pivot 10 (:531); log with scalingFactor 1 (:356) is always 0.0, because 1f + 4.5918E-41f is 1f. Saturation without a pivot takes the pivot from the index statistics (:408-415, :725-738), which on a one-document MemoryIndex decode to the same 4.5918E-41, so it scores 0.5 as on IndexWriter; tiny explicit pivots give positive but wrong scores.
4. 0.0 is "no match". MemoryIndex.search returns "0.0 indicating no match" (MemoryIndex.java:1105-1106; the class example tests score > 0.0f, :153-157), and Monitor's ScoringMatch reports a match only when score > 0 (ScoringMatch.java:37). That convention was already imperfect: a FILTER-only query and a boost-0 query match but score 0.0 on both paths (executed); this bug adds matching documents that score 0.0 only because their frequency was dropped.
5. The hit set is unchanged for the tested queries. Every term is still indexed for the document. The follow-up's probe ran 12 query shapes (five FeatureField function settings, two term queries, FILTER and two MUST_NOT combinations, two zero-score controls): IndexSearcher.count was the same on both paths for all 12 (executed). A query that itself depends on frequencies could still differ; none was tested.

• Monitor. With ScoringMatch and FeatureField queries, a single document scores differently from the same document in a batch (10.0 vs 4.5918E-41), and a stored query whose only scoring parts round to 0.0 (saturation or sigmoid with an explicit pivot such as 10, log with scalingFactor 1) is not reported at all for a single document (executed: saturation on main and 10.5.2; sigmoid and log in the fact checks' Monitor runs on main). The no-pivot saturation query keeps its match (0.5 on both paths). Monitor matches a single document on MemoryIndex only since 8.6.0 (LUCENE-9314): in 8.2.0 to 8.5.2 match(Document) went through the IndexWriter-based batch (read).
• Direct MemoryIndex.search. Its javadoc makes score > 0.0f the match test; the same query shapes report "no match" for a matching document (executed).
• Custom-frequency text fields. Postings frequency, totalTermFreq, sumTotalTermFreq, the norm and BM25 scores are wrong for DOCS_AND_FREQS fields fed by DelimitedTermFrequencyTokenFilter or a custom stream, and for DOCS_AND_CUSTOM_FREQS fields (10.5.0+) (executed).
• Default vs opt-in. Default MemoryIndex, Monitor and similarity settings; the trigger is the opt-in custom-frequency feature.
• Scale. Any document, deterministic, every time.
• Silent. No exception, and for the tested queries no change in which documents match, so only code that reads frequencies, term statistics, norms or scores notices (including score-based match tests); no report was found in the nine years since 7.0.0, so the registry rates the likelihood "rare".

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<10.0> but was:<4.5918E-41>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:577)
       org.junit.Assert.assertEquals(Assert.java:701)
       org.apache.lucene.index.memory.TestMemoryIndexFeatureField.testLinearQueryScoresFeatureValue(TestMemoryIndexFeatureField.java:28)

• The attribute's contract. TermFrequencyAttribute.java:24-26: "Sets the custom term frequency of a term within one document. If this attribute is present in your analysis chain for a given field, that field must be indexed with IndexOptions#DOCS_AND_FREQS." FeatureField and the DOCS_AND_FREQS repro field meet that precondition word for word. DOCS_AND_CUSTOM_FREQS (10.5.0) is newer than that sentence and not named in it; IndexOptions.java:53-54 calls it "Like DOCS_AND_FREQS except the "custom freqs" are treated as scores", and IndexWriter honours the attribute for it, so it is a parity case. DelimitedTermFrequencyTokenFilter states the same precondition: "DOCS_AND_FREQS but no positions or offsets" (DelimitedTermFrequencyTokenFilter.java:31-32).
• FeatureField's contract. FeatureField.java:47: "Feature values are internally encoded as term frequencies." Its query functions read the value back from the frequency.
• MemoryIndex's role. MemoryIndex.java:106: "a replacement/substitute for RAM-resident Directory implementations"; :129: "Arbitrary Lucene queries can be run against this class". Monitor swaps it in for a one-document IndexWriter index only for speed (DocumentBatch.java:101-102: "MemoryIndex benchmarks as better performing than RAMDirectory for this case"), since 8.6.0, when LUCENE-9314 fixed "Single-document monitor runs were using the less efficient MultiDocumentBatch implementation" (CHANGES.txt:5804-5805).
• Tests. TestMemoryIndexAgainstDirectory duels MemoryIndex against an IndexWriter index: equal norms (:200-207), equal hit counts (:259-268), and in testDuelMemoryIndexCoreDirectoryWithArrayField equal term-vector frequencies, positions and offsets (:714-800). Its postings and sumTotalTermFreq checks (:210-250) compare the MemoryIndex reader with its own term vectors, not with IndexWriter (:193-196). Its analyzers never set the attribute. No test in lucene/memory or lucene/monitor uses TermFrequencyAttribute or FeatureField, and none asserts frequency 1.
• History. LUCENE-7854 (d276acfbbcb8, 2017) added the attribute in 14 files, none in lucene/memory; #15584 (DOCS_AND_CUSTOM_FREQS, 10.5.0) changed one MemoryIndex line without adding support. Neither discussion mentions MemoryIndex or Monitor.
• Precedents. LUCENE-7679 / #8730 (type:bug, Bug Fixes in 6.5.0, CHANGES.txt:8442-8443): "MemoryIndex was ignoring omitNorms settings on passed-in IndexableFields", which also made its scores differ from IndexWriter's. LUCENE-8756 / #9801 (type:bug): "MLT queries ignore custom term frequencies", the same omission in MoreLikeThis; 8.1.0's CHANGES lists the fix under "Changes in Runtime Behavior" (CHANGES.txt:6564, entry :6578-6579).

The strongest works-as-designed case, built by the verifiers and a reading-only review by a model from another family (rated "moderate"): MemoryIndex does not mirror index options. createFieldInfo always declares DOCS_AND_FREQS_AND_POSITIONS[_AND_OFFSETS] (MemoryIndex.java:759-763), TestMemoryIndex.testKeywordWithoutTokenStream (:1093) accepts occurrence counts with positions for DOCS-only keyword fields, and IndexWriter rejects custom frequencies together with positions ("cannot index positions while using custom TermFrequencyAttribute", FreqProxTermsWriterPerField.java:208-219). The attribute's javadoc names only DOCS_AND_FREQS, and Monitor's use of MemoryIndex is a speed optimization, not a stated promise of full equivalence. So custom frequencies may never have been part of MemoryIndex's model.

Why it fails: MemoryIndex accepts these stock fields without complaint and then returns wrong scores, including a documented "no match" for a matching document. The caller met the attribute's precondition; it is MemoryIndex that overrides the index options. Nothing in javadoc, CHANGES, MIGRATE or tests documents a limitation (compare GITHUB#15334, whose CHANGES entry says "MemoryIndex does not support nearest neighbor search"). Honouring the frequency or rejecting the field would both remove the silent wrong score. The argument does hold for addField(String, TokenStream), which always indexes positions (it uses defaultFieldType, MemoryIndex.java:476-479, :728; its javadoc does not say so).

One fix was applied in a scratch worktree at a9779ef3c193 and run through gradle (seed DEADBEEF); the other options were not run. The hunt's first sketch (no overflow check) is listed last because it must not be used as is.

storeTerms gets the field type's IndexOptions. For DOCS_AND_FREQS and DOCS_AND_CUSTOM_FREQS it reads TermFrequencyAttribute and adds the frequency to the postings frequency, maxTermFrequency, sumTotalTermFreq and the field length (1 per term for DOCS_AND_CUSTOM_FREQS, as IndexingChain does). Positional field types, including every addField(String, ...) path and the highlighters' re-analysis, keep today's one-per-token counting. Int sums go through Math.addExact and throw IllegalArgumentException("too many tokens for field ...") like IndexingChain. MemoryPostingsEnum.nextPosition returns -1 once the stored positions run out (a frequency of 33344 has one stored position), which is what PostingsEnum returns when positions are not indexed. A second diff adds the repro tests plus testCustomFrequencyOverflowIsRejected.

final boolean customFreqs =
    indexOptions == IndexOptions.DOCS_AND_FREQS || indexOptions == IndexOptions.DOCS_AND_CUSTOM_FREQS;
final boolean termDoc = indexOptions == IndexOptions.DOCS_AND_CUSTOM_FREQS;
TermFrequencyAttribute termFreqAtt =
    customFreqs ? stream.addAttribute(TermFrequencyAttribute.class) : null;
// per token:
final int termFreq = termFreqAtt == null ? 1 : termFreqAtt.getTermFrequency();
info.numTokens = addTokens(info, info.numTokens, termDoc ? 1 : termFreq); // was ++
info.sliceArray.freq[ord] = addTokens(info, info.sliceArray.freq[ord], termFreq); // was ++
info.sumTotalTermFreq += termFreq; // was ++

Trade-offs

• The positional view stays inconsistent. MemoryIndex still declares positions (hasPositions true) while a FeatureField posting now has frequency 33344 and one stored position: later positions read as -1, term vectors report frequency 33344, and toStringDebug prints "positions=33344" next to one real position (executed). -1 is what PostingsEnum returns when positions are not indexed, but this reader says they are, so a positional query on a FeatureField runs instead of throwing as on an IndexWriter index. Making createFieldInfo follow the field type would fix it, but changes what testKeywordWithoutTokenStream asserts.
• Not validated for position consumers. No test runs a phrase, span or other positional query, or CheckIndex, against a fixed custom-frequency field; apart from this doc's own tests the upstream memory and monitor suites have no FeatureField or TermFrequencyAttribute case. A PR should add such tests, or stop advertising positions for these fields.
• Positional fields stay lenient. A TextField analysed with DelimitedTermFrequencyTokenFilter makes IndexWriter throw "cannot index positions while using custom TermFrequencyAttribute"; MemoryIndex keeps counting 1. Throwing there too could break highlighting of such fields, which re-analyses text into a MemoryIndex through addField(String, TokenStream) (MemoryIndexOffsetStrategy.java:112, WeightedSpanTermExtractor.java:411-413).
• DOCS fields stay lenient. For a repeated term with a frequency other than 1, IndexWriter throws "must index term freq while using custom TermFrequencyAttribute" (FreqProxTermsWriterPerField.java:147-152); the fix does not read the attribute for DOCS, so MemoryIndex accepts it.
• No duplicate-term check for DOCS_AND_CUSTOM_FREQS. IndexWriter rejects the document with "Document update skipped due to duplicate termdoc term" (IndexingChain.java:2095-2097, caused by "has duplicate term" at FreqProxTermsWriterPerField.java:193-194); MemoryIndex adds the two frequencies (executed). The hunt's harness counted 432 such cases; a separate parity gap that can go in the same PR.
• The overflow exception leaves the MemoryIndex partly updated, like any exception in the middle of a TokenStream (IndexWriter marks the document deleted). Not benchmarked: per token one attribute read and two checked additions.

What the checks validated

• Stock main a9779ef3c193: the memory repro fails 4 of 5 and the Monitor test 2 of 2, with assertions on and off; the control passes.
• With the fix: the 5 memory tests plus testCustomFrequencyOverflowIsRejected and the 2 Monitor tests pass, with assertions on and with assertions off.
• With the fix, all of lucene/memory, lucene/monitor and lucene/highlighter and core's TestFeatureField and TestCustomTermFreq pass: 73 classes, 991 tests, 1 skipped, 0 failures.
• ecjLint and checkGoogleJavaFormat pass for memory and monitor, and forbiddenApis for memory, with the tests in place.
• Through the classpath runner with the fixed MemoryIndex.java shadowed, the finder's repro (3 of 3) and the independent reproducer's (7 of 7) pass; the follow-up's input of 50,000 tokens with frequency Integer.MAX_VALUE, which made the field vanish under the hunt's first sketch, now throws "too many tokens for field".

Throw IllegalArgumentException when a token's frequency is not 1 ("MemoryIndex does not support custom term frequencies"). Not run.

Trade-offs

Small and removes the silent wrong score, but every document with a FeatureField that goes through a MemoryIndex (a single-document Monitor match, the percolator path) would start failing, even when no query reads the field; Monitor batches of two or more use IndexWriter and are not affected. Given MemoryIndex's always-positional design, maintainers may well prefer this option.

Say in the MemoryIndex class javadoc that custom term frequencies are counted as 1. (GITHUB#15334 stated its kNN limitation only in its CHANGES entry, CHANGES.txt:1476-1477; MemoryIndex.java has no such javadoc.) Not run.

Trade-offs

Makes the behaviour a documented limitation, but leaves the wrong scores and the "no match" for matching documents in place.

Reads the attribute for every field type and adds it with plain +=. It passes the repros and the 51 upstream memory tests.

Trade-offs

With 50,000 tokens of frequency Integer.MAX_VALUE the field length wraps and the field vanishes from the reader (MemoryIndex hides fields with numTokens <= 0, MemoryIndex.java:1834, 1842), executed; with the recommended fix the same input throws "too many tokens for field".

Method: bounded-exhaustive translation validation on the stock jar (semi-formal tier), in a hunt unit on MemoryIndex equivalence.

• The harness builds each case twice, with a one-document IndexWriter index and with MemoryIndex, and compares the postings frequency, totalTermFreq, sumTotalTermFreq, the norm and query scores through LeafReader and the query API. Scope CF: one field, DOCS_AND_FREQS and DOCS_AND_CUSTOM_FREQS, omitNorms on and off, every stream of at most two tokens over {a, b} with custom frequencies {1, 2, 7}, plus FeatureField with 4 values and 4 query functions: 1,836 cases, 4,220 comparisons.
• Mismatches in every class (for example 1,120 DOCS_AND_FREQS and 800 DOCS_AND_CUSTOM_FREQS score mismatches, and all 16 FeatureField query scores); 0 mismatches with the fix sketch shadowed.
• Found 2026-10-26 on main ea57360912ed; then turned into public-API JUnits and checked by an independent reproducer, an intended-behaviour devil's advocate, reading-only reviews by an independent reviewer from another model family (OpenAI GPT-5.6), an adjudication and a follow-up with history and prior-report searches.