Block-tree TermsEnum.next() after seekExact(term, TermState) on a new enum restarts at the field's first term

TermsEnum.seekExact(BytesRef, TermState) puts an enum on a term from saved state; the block-tree terms dictionaries do this without re-seeking the dictionary (the javadoc allows it for low-level implementations; the generic BaseTermsEnum just seeks the term). TermStates.build and TermQuery.getTermStates() hand out such states, and TermQuery applies them to a new enum of each segment. With the block-tree terms dictionary (the default codec, BlockTreeOrds and the Lucene90 block tree used for 9.x-10.2 indexes), when that seek is the first call on a new enum, term() and docFreq() are right but next() returns the field's first term and the enum then lists the whole field again. The seek leaves the lazily cloned input null, and next() reads a null input as "fresh enum" before its own catch-up branch for TermState seeks can run. Three one-term documents a, b, c: after the seek to b, next() returns a instead of c, silently, with assertions on and off (main and 10.6.0). Every release from 4.0.0-alpha to 10.6.0 has the same code order, and every release that was run fails the same way. No in-tree Lucene query calls next() after a TermState seek and no wrong search result is known; code that drives the Expert TermsEnum API directly is affected.

1. The SegmentTermsEnum constructor sets currentFrame = staticFrame and does not clone the terms input (SegmentTermsEnum.java:60-69, the root-block load at :68-69 commented out); initIndexInput() clones it lazily (:81-85), when a frame first prefetches or loads a block (SegmentTermsEnumFrame.java:147, :167).
2. seekExact(BytesRef, TermState) (:1027-1046) sets currentFrame = staticFrame (:1035), copies the state and the term, and loads no block, so in stays null.
3. next() (:892) first tests if (in == null) { // Fresh TermsEnum; seek to first term (:893) and makes the root frame the current frame in place of staticFrame (:896).
4. Only then comes if (currentFrame == staticFrame) (:911-922), whose comment says that after seek(TermState), "if they then call next(), this method catches up all internal state so next() works properly". It re-seeks the term (:920), but step 3 already replaced currentFrame, so it never runs and next() returns the first term of the root block.
5. The same order is in BlockTreeOrds (OrdsSegmentTermsEnum.java:906 before :932) and in the Lucene90 block tree in backward-codecs (SegmentTermsEnum.java:983 before :1008); the sandbox IDVersionSegmentTermsEnum has it too (next() at :1016, its fresh-enum branch at :1018-1019).

• Terms a, b, c, seek to b: term() is b and docFreq() 1, then the next() calls return a, b, c, null instead of c, null. A seek to a gives a, b, c, null (expected b, c, null); a seek to c gives a, b, c, null (expected null).
• Terms kiwi, lemon, mango, olive, peach: for every one of the five targets the first next() returns kiwi; for peach, the last term, it should return null; after the seek to mango the next() calls return kiwi, lemon, mango, olive, peach instead of olive, peach.
• 3,000 terms over several blocks: targets #0, #1, #1500, #2998 and #2999 all return the first term.
• An exhaustive check of every set of 1-3 short terms (the writer set to 2 to 3 entries per block) against a sorted-array reference: 10,277 of 10,700 sets mismatch, every mismatch a next() after the TermState seek.
• Not affected: a state taken from the same enum, an enum positioned once by an ordinary seek or next() first, and term(), docFreq(), totalTermFreq(), postings() and impacts() right after the seek (a probe asserts all five on the default block tree, BlockTreeOrds and the Lucene90 block tree).

• Who is hit. Application or library code that drives TermsEnum directly: it positions a brand-new block-tree enum with seekExact(term, state), the state taken from TermStates, TermQuery.getTermStates() or a saved (term, state) pair, and then calls next(), for example to list the terms after a known term or to resume term paging. Such code gets wrong terms every time.
• What it does. A paging loop that opens a new enum for each page and resumes it from the previous page's last (term, TermState) makes no progress, because each resumed page starts again at the field's first term: capped at 200 pages of 100 it delivered 20,000 terms for a field with 2,706 distinct terms and had not finished; with the fix it finished in 28 pages. A listing that starts at a TermState and stops at an upper bound also returns every term below its start. The wrong listing stays inside the same field and segment and ends after one extra pass.
• Who is not. Lucene's queries. Every in-tree caller of the TermState seek reads only postings, impacts or statistics afterwards: outside sandbox TermQuery, SynonymQuery, PhraseQuery, MultiPhraseQuery, CombinedFieldQuery, the multi-term constant-score wrappers and SpanTermQuery; in sandbox PhraseWildcardQuery, TermAutomatonQuery and the sandbox CombinedFieldQuery. FuzzyTermsEnum and FilterLeafReader only pass the call on to another enum. 57 queries over the paths outside sandbox, on an index with deletions and with the bug active in 4 of 4 leaves, matched a SimpleText oracle exactly in all hits and their scores, the total hit counts, count(), and the explanation's match flag and value for the top three hits of each query. The downstream code read (Elasticsearch, OpenSearch and its security plugin, Solr; default branches, not exhaustive) has no caller that calls next() after a TermState seek. So no wrong Lucene search result is known; this covers today's in-tree callers, not future or downstream code.
• Default vs opt-in. Default codec and every block-tree terms dictionary; the trigger is an Expert call sequence on an @lucene.experimental class (TermsEnum.java:37).
• Scale. One segment and three terms are enough; deterministic.
• How likely. Rare: the recorded search found no public report since 4.0 (2012), although a 2011 dev@ thread shows a user calling seek(BytesRef, TermState) followed by next(), before block tree existed.

The test indexes a, b and c in field f (one segment, ByteBuffersDirectory, default codec pinned with TestUtil.getDefaultCodec(), which returns new Lucene104Codec(), not the test framework's randomized codec), takes the state of f:b with TermStates.build(new IndexSearcher(reader), term, false).get(leaf).get(), applies it to a new enum from leaf.reader().terms("f").iterator() and asserts that next() returns c. Deterministic, single thread, no random data. On 9.x, TermStates.get(leaf) returns the TermState itself: drop the trailing .get(); that port fails the same way on 9.12.3.

TestNextAfterSeekExactTermState > testNextAfterSeekExactWithTermStateOnNewEnum FAILED
    org.junit.ComparisonFailure: expected:<[c]> but was:<[a]>

# the same with -Ptests.asserts=false; with the recommended fix the test passes

• Order and position. TermsEnum.java:31-32: "Term enumerations are always ordered by BytesRef.compareTo ... Each term in the enumeration is greater than the one before it." :34-35: the enum is unpositioned until you "successfully call next or one of the seek methods". :98: "Expert: Seeks a specific position by TermState".
• The code promises it. The comment in next() (SegmentTermsEnum.java:912-917) says that if "seek(TermState) was called ... if they then call next(), this method catches up all internal state so next() works properly". The catch-up exists; it is unreachable on a new enum.
• Lucene uses cross-instance states itself. The NOTE in the same javadoc (TermsEnum.java:105-108) calls states compatible "only iff the source and the target TermsEnum operate on the same field", which only makes sense for two enums. TermStates.build/get and TermQuery.getTermStates() are public and return states whose source enum the caller never sees; TermQuery applies them to a new enum (TermQuery.java:140-141), and so does the test framework's RandomPostingsTester (:1613, :1628).
• The test framework's state model accepts it. AssertingTermsEnum marks the enum POSITIONED after seekExact(term, state) (AssertingLeafReader.java:454-460) and accepts next() from there (:332-343). It does not check what next() returns, so this shows how Lucene's own wrapper models the sequence, not what the javadoc promises.
• Every other dictionary returns the successor. After the same calls (seek to banana), FST50, Direct, BlockTerms (two index variants), MemoryIndex and SimpleText return cherry; only the default block tree and BlockTreeOrds return apple. A later fact check found the same for UniformSplit and STUniformSplit.
• History. On unreleased trunk before LUCENE-3562 (2011) the constructor loaded the root block, so next() reached the catch-up. LUCENE-3562, a performance change, made the input clone lazy and put the "fresh" branch first; nothing in it discusses next(). Stock 4.0.0 fails and 4.0.0 with only that constructor change reverted passes. In 2013 mikemccand wrote on LUCENE-3069 (#4142) that the lazy TermState seek is by design and "We only need to re-load the metadata if the caller then tries to do .next()".

The strongest case for "works as designed": TermsEnum.java:102-103 says "Seeking by TermState should only be used iff the state was obtained from the same TermsEnum instance." Following it, the bug cannot happen, because taking a state from an enum loads a block; the only upstream test that calls next() after a TermState seek uses the same enum (TestTermsEnum.java:686); and every in-tree cross-instance caller pulls only postings or statistics. A cross-model reviewer called the trigger out of contract on this sentence alone.

Why it is weaker than it looks, though not settled: the sentence is hard to reconcile with Lucene's own use of states. TermStates.build and TermQuery.getTermStates() hand out states built from enums their callers never see, so those states can never meet the sentence, and TermQuery itself applies them to a new enum (a caller can also put its own state into a TermStates through its public constructor or register). A maintainer may still read the sentence as a precondition for calling next() after such a seek, with cross-instance states meant only for postings and statistics. The NOTE right after the sentence speaks of a "source" and a "target" enum and warns that an incompatible state "might leave this TermsEnum in undefined state"; this state meets its compatibility condition (same field, same segment), but the NOTE does not promise that next() works after a seek with another enum's state. The sentence says "should", not "must"; the next() comment promises the opposite; and the sentence came from a 2013 commit titled "javadocs" with no issue or discussion, which replaced the 2011 wording "obtained from the same IndexReader". The same day, 47 minutes before that commit, mikemccand had written on LUCENE-3069 (#4142) that the lazy TermState seek "is normal / by design" and "We only need to re-load the metadata if the caller then tries to do .next()"; only the timestamps are recorded, no link between the two is claimed. Under that wording the repro is in contract, and it already failed on 4.0.0-ALPHA and 4.0.0. None of this settles the question; the maintainers decide it. If they hold the sentence as binding, the outcome is a clearer javadoc and a corrected next() comment; either way the docs and the code disagree today. Our reading: not intended, a defect. The adjudication's confidence is medium because of this sentence, and the maintainers choose between the code fix (A) and a javadoc fix (C).

Option A was applied in scratch worktrees of 645d4fe14a20 and run with gradle (each worktree removed after the runs): once for the fact-check resolution of this page's first version, and again, fresh, for this version. It was revised after the fact checks: its first version never cleared the flag (see the trade-offs). Options B and C were not built.

In the lucene103 SegmentTermsEnum, the Lucene90 copy in backward-codecs and OrdsSegmentTermsEnum: a seekedByTermState field, set in seekExact(BytesRef, TermState), cleared at the start of the other seeks (seekExact(BytesRef) and its two-phase form, seekCeil, and in OrdsSegmentTermsEnum seekExact(long)), and tested in next(). next() then falls through to its existing catch-up, which re-seeks the term and loads its block: the same path an enum takes today when it was positioned once before the TermState seek.

// new field
private boolean seekedByTermState;

// seekExact(BytesRef) (via prepareSeekExact) and seekCeil
seekedByTermState = false;

// next()
if (in == null && seekedByTermState == false) {
  // Fresh TermsEnum; seek to first term:
  ...
}

// seekExact(BytesRef, TermState)
currentFrame = staticFrame;
seekedByTermState = true;

Trade-offs

• Keeps the lazy clone of LUCENE-3562: a TermState seek followed only by postings or statistics, the TermQuery path, still clones nothing.
• One boolean per enum, written by each seek; the first next() after a TermState seek pays one term lookup, as an enum positioned before the seek does today. No benchmark was run.
• Why the other seeks clear the flag: the first version of this fix never cleared it (and reused the positioned flag in OrdsSegmentTermsEnum). That changed a sequence outside the contract, a TermState seek, then a seekExact(BytesRef) that returns false without loading a block, then next() on the now unpositioned enum (AssertingTermsEnum rejects that next()): both SegmentTermsEnum copies then continued after the old TermState term instead of restarting like a new enum, or threw (AssertionError, and ArrayIndexOutOfBoundsException without assertions) when the failed seek ended on a block without terms. The cross-model fact check pointed to this path; with the flag cleared, the sequence behaves as on stock.
• The sandbox IDVersionSegmentTermsEnum has the same order and is not patched.

What the checks validated

• This version's own run (fresh scratch worktree, seed DEADBEEF): stock fails; with this fix the repro, two scratch probes of its steps on BlockTreeOrds and the Lucene90 block tree, and core TestTermsEnum, TestTermsEnum2, TestTermStates, TestTermQuery, TestLucene104PostingsFormat, codecs TestOrdsBlockTree and backward-codecs TestLucene90PostingsFormat give 129 tests (126 upstream), 0 failures, 7 @Nightly skipped; with assertions off the repro and TestTermsEnum give 14 tests, 0 failures, 1 @Nightly skipped.
• Scratch probes, 7 tests each on the default block tree, BlockTreeOrds and the Lucene90 block tree (statistics, postings and impacts right after the TermState seek; next() after postings; two TermState seeks; seekCeil; three failed seeks compared with a new enum). Stock: the two next() tests fail in all three (the bug). First fix version: the three failed-seek tests fail in both SegmentTermsEnum copies. This fix: all 21 pass.
• Gradle with this fix, seed DEADBEEF: the repro, the 21 probe tests, two probes of the repro's steps on BlockTreeOrds and the Lucene90 block tree, and core TestTermsEnum, TestTermsEnum2, TestLucene104PostingsFormat, TestTrie, TestTermStates, TestTermQuery, codecs TestOrdsBlockTree and backward-codecs TestLucene90PostingsFormat, TestLucene99PostingsFormat, TestLucene101PostingsFormat, TestLucene101PostingsFormatV0: 251 tests (227 upstream), 0 failures, 14 @Nightly skipped, all upstream, so 213 upstream tests ran.
• With assertions off: the repro, the probes, TestTermsEnum, TestOrdsBlockTree and TestLucene90PostingsFormat: 106 tests (82 upstream), 0 failures, 5 @Nightly skipped.
• checkGoogleJavaFormat of the core, codecs and backward-codecs modules passes with the fix.
• Earlier, the first fix version passed 230 of these tests, and the finder's two-file version of it passed an exhaustive harness (0 mismatches over 204,180 term sets, and over 9,196,956 sets in a larger run), a seven-dictionary differential, an independent verifier's 8 tests and 83 upstream tests. This fix was not run through the exhaustive harness.

Call initIndexInput() in the TermState seek, so next() never sees a null input after it.

Trade-offs

One line per copy, but it brings back the per-seek input clone that LUCENE-3562 removed on purpose, on the path every TermQuery takes for every segment. Not run, so it is unknown whether the catch-up handles an enum that has an input but no pushed frames.

State in the TermsEnum javadoc that next() is not supported after seekExact(term, state) unless the state came from the same enum, and correct the comment in next().

Trade-offs

No code risk, but it leaves a silent wrong listing in a public API, and states from TermStates.build or TermQuery.getTermStates() can never meet that condition. Only if the maintainers hold the "same TermsEnum instance" sentence as binding.

Method: semi-formal (bounded-exhaustive translation validation plus code reading).

• An AI finder was extending a harness that writes term sets with the real block-tree writer, reads them with the real reader and checks every TermsEnum operation against an exact reference (the sorted term array), over every set of 1-3 terms built from short strings and small block sizes.
• Its TermState walks only reused the same enum. Reading the seek code, the AI tool added a walk that takes the state of a term from one enum, applies it as the first call on a second enum, then runs 12 random operations. On the stock main jar it mismatched on 10,277 of 10,700 term sets, every mismatch a next() after the TermState seek.
• The AI tool then wrote a deterministic JUnit test that drives only public Lucene APIs, which was later reduced to the one-method test. So this was found by targeted code reading checked by an exhaustive harness, not a blind discovery by a tool.