Skipped positions are counted in an int: once one term's pending occurrences in a postings block (the skipped documents plus the current one) pass Integer.MAX_VALUE, nextPosition() returns an earlier document's positions

Lucene104PostingsReader.BlockPostingsEnum reads positions lazily. When a consumer passes documents without reading their positions, the reader only adds their frequencies to posPendingCount, the number of positions it is "behind", and skips that many on the next nextPosition(). That count, the helper that sums the frequencies (sumOverRange) and the skip distance (toSkip) are all int. When the skipped documents of one postings block plus the current one hold more than 2,147,483,647 occurrences of the term, the sum wraps negative, the skip is not done, and nextPosition() returns positions from the start of an earlier document.

With assertions off this is silent: in the repro PhraseQuery f:"v t" counts 0 instead of 1. With assertions on, assert posPendingCount > 0 fails. A default IndexWriterConfig and the default codec suffice, and every document is within Lucene's per-document limits, because the trigger is cumulative across documents. It needs huge documents, for example two that each repeat one term about a billion times, so the priority is low. The Lucene912 reader of 9.12.x and 10.0.0 used long and is correct; making the three values long again fixes it.

1. The count. private int posPendingCount;, commented "how many positions "behind" we are; nextPosition must skip these to "catch up"" (:381-383); set to 0 when the enum is reset to a term (:538).
2. The catch-up. On the first nextPosition() of a document (:1299-1301), accumulatePendingPositions() (:1263-1274) runs posPendingCount += sumOverRange(freqBuffer, posDocBufferUpto, docBufferUpto): the frequencies of every document of the doc buffer whose positions were not read, plus the current one. It then asserts posPendingCount > 0 (:1268) and calls skipPositions(freq) only if (posPendingCount > freq) (:1270).
3. All int. sumOverRange sums into int res (:270-276), and skipPositions computes int toSkip = posPendingCount - freq (:1133). In the repro, 2,147,483,649 wraps to -2,147,483,647 inside sumOverRange.
4. No skip. A negative count is never > freq, so decoding continues where the position stream stands: the start of doc 0's positions (:1306-1310). Payloads and offsets are read in lockstep from the same point (:1276-1290, code reading).
5. Exact boundary (independent checkers, executed): a sum of exactly Integer.MAX_VALUE reads correctly and Integer.MAX_VALUE + 1 does not; 2^31 - 16 is correct and 2^31 + 48 wrong.
6. One block. Moving to the next level-0 block while the position stream is behind seeks forward and resets the count from skip data (doMoveToNextLevel0Block, :721-736; seekPosData, :799-815). Only when that block's positions were already decoded does it add the remaining frequencies to the count instead (:735, :814). So from 10.1 on, on the lazy path the repro takes, the overflowing sum must in practice come from one block: 256 documents containing the term in Lucene104 (since 10.4), 128 in Lucene101/103 (code reading). The reset normally also ends the damage at the end of the block; the two carry-over branches were not exercised, so this is not a guarantee.

PhraseQuery f:"v t" lands on doc 2 directly. Sum of frequencies: 2,147,483,649, one int wrap: -2,147,483,647. The reader returns 0 for t in doc 2, the phrase needs 1, so the query counts 0. A direct-read probe on this exact index (fact check, executed) printed these values: the int running sums 1,073,741,824, -2,147,483,648, -2,147,483,647 and nextPosition() = 0 on doc 2. On main ea57360912ed the checkers' own variants printed the substituted positions directly: "first position of t in doc 1: 0 (expected 1)" and "expected first positions [101, 102, 103], got [1, 2, 3]".

• Who can be hit. Every consumer of PostingsEnum#nextPosition on the default codec: PhraseQuery (executed), MultiPhraseQuery, span and interval queries, highlighters and payload consumers (code reading). Positions are indexed by default for TextField. Old 10.1-10.3 segments are read by the backward-codecs Lucene101/103 readers, which have the same int code.
• Defaults. A default IndexWriterConfig (16 MB buffer, TieredMergePolicy), the default codec and forceMerge(1) reproduce it. No expert setting is needed.
• Scale. More than Integer.MAX_VALUE (2^31 - 1) occurrences of one term among the skipped documents of one postings block plus the current one (a pending sum of exactly 2^31 already fails): for example two documents of 2^30 occurrences (about 1 GiB of indexing buffer each; the test needs a 4 GB heap and about 2.5-5 minutes), or about 8.4 million occurrences per document across a full 256-document block (code reading). Positions forbid custom term frequencies, so every occurrence is a real token. One document alone cannot reach it, but not because of IndexWriter.MAX_POSITION, which limits position values: stacked occurrences (position increment 0) get around it, and the per-field token count would allow Integer.MAX_VALUE tokens. What stops one document is the writer's in-memory postings pool, which buffers at least one byte per occurrence and is capped at 2^31 bytes: a document with Integer.MAX_VALUE occurrences of one term fails while indexing with ArithmeticException: integer overflow at ByteBlockPool.nextBuffer (ByteBlockPool.java:199; executed by the fact check). So two large documents are the smallest input.
• What goes wrong. Assertions off: silent wrong positions for the documents after the trigger documents in that block (executed: a phrase false negative). Code reading only, not executed: false positives when the substituted positions happen to line up; offsets and payloads taken from the earlier, possibly deleted, document; and a merge that skips deleted documents without reading their positions could write wrong positions into the merged segment. Assertions on: an AssertionError from that nextPosition() call.
• Bounds (code reading): reads stay inside the same term's position data, and the block reset normally ends the damage at the end of the block (the carry-over branches were not exercised); no hang and no memory unsafety, and with the skip dropped the reader does less work, not more.

The test sits in org.apache.lucene.search and uses only public API. It pins the default codec because LuceneTestCase randomizes it, and writes to an FSDirectory in a temporary directory (about 1-2 GB of disk while it runs). Each 2^30-token document flushes to its own segment under the default 16 MB buffer, and the forced merge orders segments by size, largest first, so v t ends up last; the test asserts only the phrase count, not the segment count, that order or the raw positions (probes on the same index printed those: one leaf, doc order 2^30, 2^30, v t, nextPosition() = 0 on doc 2). Two large documents are the smallest input: one document with Integer.MAX_VALUE occurrences of one term already fails while indexing, because the writer's postings pool, at least one byte per occurrence, is capped at 2^31 bytes (ByteBlockPool.java:199; IndexWriter.MAX_POSITION limits only position values, which stacked tokens get around).

As a regression test it is the same test; it fails on stock and passes with the fix. Its scale matches the @Monster tests, like Test2BPositions (which run only with -Ptests.monster=true). All runs were on Linux aarch64 with OpenJDK 25.0.3; the mechanism is plain int arithmetic.

# gradle, stock 645d4fe14a20, seed DEADBEEF, assertions on
TestPhraseQueryPendingPositionsOverflow > testSkippedFreqsPastIntMax FAILED
    java.lang.AssertionError
        at org.apache.lucene.codecs.lucene104.Lucene104PostingsReader$BlockPostingsEnum.accumulatePendingPositions(Lucene104PostingsReader.java:1268)
        at org.apache.lucene.codecs.lucene104.Lucene104PostingsReader$BlockPostingsEnum.nextPosition(Lucene104PostingsReader.java:1301)
        at org.apache.lucene.search.ExactPhraseMatcher.advancePosition(ExactPhraseMatcher.java:141)

# assertions off (-Ptests.asserts=false)
TestPhraseQueryPendingPositionsOverflow > testSkippedFreqsPastIntMax FAILED
    java.lang.AssertionError: hits of f:"v t" (only the last doc matches; the 3 freqs of t sum to 2^31 + 1) expected:<1> but was:<0>

# with the fix, assertions on and off
:lucene:core:test (SUCCESS): 1 test

# release jars, assertions off: 10.6.0 fails (expected:<1> but was:<0>), 10.0.0 passes

• The contract. "Returns the next position, or -1 if positions were not indexed. Calling this more than freq() times is undefined." (PostingsEnum.java:85-88). The repro calls it once for doc 2. Skipping other documents' positions is the normal lazy pattern, and PhraseQuery itself relies on it.
• The documented limits are per document. IndexingChain rejects a position above IndexWriter.MAX_POSITION (IndexingChain.java:2021-2028) and a field length that overflows an int, "too many tokens for field" (Math.addExact, :2054-2064). Each repro document has 2^30 tokens. Test2BPositions indexes "> Integer.MAX_VALUE positions" for one term (Test2BPositions.java:35), and the Lucene104 format's "Limitations" section names only int document numbers (package-info.java:426-436).
• The code's own assumption. assert posPendingCount > 0 (:1268) says the count is never negative; it fires on this input.
• History. The Lucene912 reader (9.12.x, 10.0.0, #13585) declared long posPendingCount, long toSkip and a long sumOverRange (backward-codecs Lucene912PostingsReader.java:302, :629, :939) and reads the same index correctly (executed on 9.12.0, 9.12.3 and 10.0.0). #13968 returned to int; its description and reviews never mention overflow.
• Precedent. LUCENE-4468 / #5534, "Fix rareish integer overflows in Lucene41 postings format" (CHANGES.txt:14734), fixed the same class of bug in file-pointer fields.

The strongest case for "intended": the scale is absurd, since nobody puts 2^31 occurrences of one term into 256 documents, and the int[] move in #13968 was a deliberate performance choice for more SIMD lanes.

Why it fails: the scale explains the low priority but is not a documented limit, and every input passes Lucene's own validation with default settings. The SIMD motivation covers the int[] buffers, not a scalar counter or a scalar sum, and 9.12.x and 10.0.0 shipped the long version. The checkers that gave a strength rating (the independent reproducer and reviews by another model family, OpenAI GPT-5.6) rated the intended-behaviour argument weak; the devil's advocate also rejected it, without a rating.

The hunt's fix, extended to the two backward-codecs readers that share the code, was run through gradle in a scratch worktree of 645d4fe14a20 (OpenJDK 25.0.3, seed DEADBEEF).

In Lucene104PostingsReader (:270-276, :383, :1131-1174) and identically in the backward-codecs Lucene101PostingsReader and Lucene103PostingsReader: the types the Lucene912 reader used. The casts sit where toSkip is already below the 256-entry block (toSkip < leftInBlock, or after the while (toSkip >= BLOCK_SIZE) loop); payloadByteUpto += sumOverRange(...) keeps compiling through compound-assignment narrowing, and that sum stays within one block.

-  private static int sumOverRange(int[] arr, int start, int end) {
-    int res = 0;
+  private static long sumOverRange(int[] arr, int start, int end) {
+    long res = 0;
-    private int posPendingCount;
+    private long posPendingCount;
-      int toSkip = posPendingCount - freq;
+      long toSkip = posPendingCount - freq;
-        int end = posBufferUpto + toSkip;
+        int end = posBufferUpto + (int) toSkip;
-          payloadByteUpto = sumOverRange(payloadLengthBuffer, 0, toSkip);
+          payloadByteUpto = (int) sumOverRange(payloadLengthBuffer, 0, (int) toSkip);
-        posBufferUpto = toSkip;
+        posBufferUpto = (int) toSkip;

Trade-offs

• The int[] buffers that motivated #13968 are untouched; one scalar field and one reduction over at most 256 ints become long. No benchmark was run.
• The backward-codecs Lucene90 and Lucene99 readers have the same int counters in three enums each (with a long[] freq buffer); the analogous change there was not written or run.

What the checks validated

• Stock: the repro fails with assertions on and off.
• Fix: the repro passes with assertions on and off.
• The closest core suites (codecs.lucene104.* with TestLucene104PostingsFormat, and the positions, payloads, offsets, skip-list and phrase tests): 24 classes, 278 tests, 0 failures, 7 skipped.
• The whole core suite with the repro in the tree: 749 classes, 8,898 tests, 0 failures, 322 skipped.
• backward-codecs lucene101.* and lucene103.*: 106 tests, 0 failures, 6 skipped; queries spans.* and intervals.*: 349 tests, 0 failures, 2 skipped.
• :lucene:core:check and :lucene:backward-codecs:check with -x test pass (google-java-format, ecjLint, forbiddenApis, checksums).
• Earlier, shadowed onto the stock jars, the Lucene104 change passed the finder's repro, the refute reviewer's own check and the minimal test.

A long sumFreqs(int[], int, int) for the three posPendingCount += sites (:735, :814, :1265), with posPendingCount and toSkip long.

Trade-offs

Same effect; a slightly larger diff that leaves the payload-length path unchanged.

Math.addExact on the count.

Trade-offs

Turns silent wrong results into an exception on an index that Lucene accepted and that a long reads correctly.

Cap a term's occurrences per postings block.

Trade-offs

The writer cannot know which documents a reader will skip, and a cap would reject indexes that 9.12.x and 10.0.0 read correctly.

Method: non-formal (code reading plus an executed test; no model or solver produced the signal), a hunt unit on the Lucene104 postings reader.

• The unit built a postings differential harness (0 mismatches at ordinary frequencies, 8 reader mutants killed). While modelling the reader for it, the finder compared the position bookkeeping with the Lucene912 reader and noticed int sumOverRange and int posPendingCount against long.
• A main() on the stock main jar with two documents of 2^30+1 and 2^30+2 occurrences confirmed it with assertions on and off; it used an expert RAM setting and LogDocMergePolicy.
• An independent verifier then wrote its own JUnit with a default IndexWriterConfig, which the one-test repro here is cut down from.