docIDRunEnd() on a two-phase disjunction's approximation rewrites the scorer's list of matching clauses, so score() adds a non-matching clause or never returns

The finding in 30 seconds

A disjunction scorer keeps the clauses that match the current document in a linked list, and a call on its approximation that is documented not to move it rewrites those links.

Background: two-phase disjunctions and docIDRunEnd()

Some queries check a document in two phases. A phrase query first finds the documents that contain all its terms (its approximation), then matches() checks that the terms are adjacent and in order. An OR query (a disjunction) with such a clause is two-phase too: its approximation is the union of its clauses' approximations.
Once matches() says yes, the disjunction scorer remembers which clauses really match the current document, so that score() adds up exactly their scores and getChildren() lists exactly them.
docIDRunEnd() asks an iterator where the run of consecutive matching documents that starts at the current one ends, so that bulk scorers can handle whole ranges at once. It is documented not to affect the iterator's position.

Diagram of the bug: a scorer for f:c OR "a b" on doc 0 of {0: "b a c", 1: "c"}. After matches(), the list of verified matches holds only f:c (f:c.next = null). docIDRunEnd() on the approximation walks the clause heap with topList(), which sets f:c.next to the phrase, so score() walks f:c then the phrase and returns 0.59193045 instead of 0.06880058. If the phrase also matched doc 0, the list would loop back on itself and score() would never return. — The repro's two clauses and their `next` links on doc 0: after the match check the list of matching clauses is `f:c` alone; `docIDRunEnd()` re-links `f:c` to the phrase, so `score()` also adds the phrase. Select the still to open it at full size. Generated with AI, checked against the code.

The finding, step by step

Index two documents, 0: "b a c" and 1: "c", and score BooleanQuery(SHOULD f:c, SHOULD "a b") with ScoreMode.COMPLETE and a plain IndexSearcher. Only f:c matches doc 0 ("a b" is not in "b a c"); a fresh scorer gives doc 0 the score 0.06880058.
Move a second scorer to doc 0 with iterator().nextDoc() and call twoPhaseIterator().approximation().docIDRunEnd(). It returns 1 and the scorer stays on doc 0. The method is documented not to affect the iterator's position, so you expect score() to give 0.06880058 again.
Instead score() returns 0.59193045, with no error, assertions on or off: the phrase, which does not match doc 0, is added in. getChildren() also lists the phrase as a second child.
Why: the match check chains the clauses that match doc 0 through the shared field DisiWrapper.next (here only f:c). docIDRunEnd() walks the clause heap with topList(), which rewrites the same field; because the cheaper phrase is the heap's top, it links the phrase behind f:c (with the costs swapped the score stays correct). If the phrase does match doc 0 (doc 0 = "a b c"), the chain becomes a loop and score() never returns.

Who is affected

Only custom code that drives a scorer itself (a BulkScorer, collector or scorer wrapper) and calls docIDRunEnd() on a two-phase disjunction's approximation between the match check and score(), when the two-phase clause is ahead of the plain clause in the heap. IndexSearcher and Lucene's own collectors and bulk scorers never make this call. Default settings; two documents suffice.

Mechanism, terms and numbers576 words

A disjunction scorer with a two-phase clause (a phrase, a doc-values range, ...) records in matches() which clauses really match the current document, by chaining them through the shared field DisiWrapper.next; score() and getChildren() later walk that chain (DisjunctionScorer.java:83-157, :178-193 at 5adff78bb168). DisjunctionDISIApproximation.docIDRunEnd() walks its clause heap with topList(), which rewrites the same fields (DisjunctionDISIApproximation.java:220-233, DisiPriorityQueue2.java:55-66). Called on twoPhaseIterator().approximation() between matches() and score(), when the two-phase clause is ahead of the plain clause in the heap, it puts a non-matching clause onto the chain: with {0: "b a c", 1: "c"} and BooleanQuery(SHOULD f:c, SHOULD "a b"), score() on doc 0 returns 0.59193045 instead of 0.06880058 (and getChildren() lists the phrase); when the phrase does match, the chain becomes a cycle and score() never returns. Assertions on and off, every release since 10.2.0 (#14363). docIDRunEnd() is documented not to affect the iterator's position; no Lucene code makes this call, so only application code that drives scorers itself is hit, and tests that go through newSearcher() cannot see it.

The mechanism (main `5adff78bb168`, `lucene/core/src/java/org/apache/lucene/search/`)

The list of verified matches. DisjunctionScorer.TwoPhase.matches() builds "the list of verified matches on the current doc" (DisjunctionScorer.java:83-84) through DisiWrapper.next: it walks approximation.topList() (:123), moves each plain clause onto verifiedMatches with w.next = verifiedMatches (:126-129), queues two-phase clauses as unverified (:136), and returns true without checking them when a plain clause matched (:141-143). In the repro the list is f:c with f:c.next = null, and the phrase is queued.
Reading it. score() and getChildren() call getSubMatches() (:178-181, :186-193), which checks the queued two-phase clauses and prepends each one that matches with w.next = verifiedMatches (:95-106, :99), then walk the list until null (DisjunctionSumScorer.java:40-46).
The relinking call (fix site). DisjunctionDISIApproximation.docIDRunEnd() walks leadIterators.topList() when the heap top is on the current doc (DisjunctionDISIApproximation.java:220-233, loop at :228). DisiPriorityQueue2.topList() writes top.next = null and, when both entries are on the same doc, top2.next = top (DisiPriorityQueue2.java:55-66); DisiPriorityQueueN.topList() relinks the same way for larger heaps (DisiPriorityQueueN.java:72-106). next is that shared field ("see #topList", DisiWrapper.java:36).
The result. In the repro the phrase is the heap's top and f:c the second entry, so the call sets f:c.next = phrase. getSubMatches() finds that the phrase does not match, and the walk f:c -> phrase adds the phrase's score and lists it as a child. When the phrase matches, getSubMatches() prepends it (phrase.next = f:c) and the list is the cycle phrase -> f:c -> phrase: score() spins in the loop at DisjunctionSumScorer.java:43 (with -ea in the assertion loop at DisjunctionScorer.java:108-116). getChildren() was not run alone in this state; by the code it spins in the same assertion loop with -ea, and with -da its loop (DisjunctionScorer.java:186-193) keeps adding children until it runs out of memory.
Heap order decides. With the clause costs swapped (the term clause on top of the heap) the relinking leaves f:c.next = null and the score is correct. A three-clause heap (f:c, f:d, "a b" on {0: "b a c d", 1: "c d", 2: "c d"}, a DisiPriorityQueueN) was run too: the call adds the phrase and drops one of the two matching term clauses (0.7906376 instead of 0.1007784), and fix B corrects it.

Exposure and likelihood310 words

Who. Code that drives scorers itself (a custom BulkScorer, collector or scorer wrapper) and calls docIDRunEnd() on twoPhaseIterator().approximation() of a positioned disjunction scorer that has a two-phase clause, after matches() and before score() or getChildren(). The query is ordinary: a disjunction with a two-phase clause (phrase, doc-values range, ...) and a plain clause on the same document, when the two-phase clause is ahead of the plain clause in the lead heap (in the repro the cheaper phrase is the top; with the costs swapped the score is correct).
Not hit. IndexSearcher's search and count paths and Lucene's own collectors and bulk scorers. DenseConjunctionBulkScorer asks a two-phase clause through TwoPhaseIterator.docIDRunEnd() (DenseConjunctionBulkScorer.java:47-53), whose default returns approximation().docID() without touching the heap (TwoPhaseIterator.java:130-132); DisjunctionScorer.TwoPhase does not override it.
Not seen through newSearcher(). No Lucene test makes this call (TestDisjunctionDISIApproximation calls docIDRunEnd() only on approximations it builds directly). Tests that go through newSearcher() could not see it: it wraps scorers in AssertingScorer, whose approximation wrapper is a FilterDocIdSetIterator without a docIDRunEnd() override (AssertingScorer.java:225-258 in the test framework), so the call returns docID() + 1 and never reaches the disjunction. A test with a plain IndexSearcher, like the repro, does see it; the regression test must use one.
Default vs opt-in. Default settings (new IndexWriterConfig(), a plain IndexSearcher; the test framework picks the codec per seed, and the numbers were the same over the five seeds run); the trigger is an expert, @lucene.experimental call pattern.
Effect. Silent: a wrong score and an extra child (with three clauses also a dropped matching child), or a score() that never returns, with assertions on and off; in that cycle getChildren() cannot return normally either (by the code). Only in-memory scorer state is affected; the search writes nothing to the index. Two documents are enough.

Minimal repro

One JUnit test method on the public API: two documents in a ByteBuffersDirectory, a plain IndexSearcher, one BooleanQuery, and one docIDRunEnd() call between nextDoc() and score(). Single thread; it fails with assertions on and off, with the same numbers over the five seeds run.

TestDisjunctionApproximationDocIdRunEnd.java (lucene/core, package org.apache.lucene.search, 1 test)63 lines

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package org.apache.lucene.search;

import java.io.IOException;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.document.TextField;
import org.apache.lucene.index.DirectoryReader;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.index.LeafReaderContext;
import org.apache.lucene.index.Term;
import org.apache.lucene.search.BooleanClause.Occur;
import org.apache.lucene.store.ByteBuffersDirectory;
import org.apache.lucene.store.Directory;
import org.apache.lucene.tests.util.LuceneTestCase;

public class TestDisjunctionApproximationDocIdRunEnd extends LuceneTestCase {

  public void testDocIdRunEndDoesNotChangeScore() throws IOException {
    try (Directory dir = new ByteBuffersDirectory()) {
      try (IndexWriter w = new IndexWriter(dir, new IndexWriterConfig())) {
        for (String text : new String[] {"b a c", "c"}) {
          Document doc = new Document();
          doc.add(new TextField("f", text, Field.Store.NO));
          w.addDocument(doc);
        }
      }
      try (DirectoryReader reader = DirectoryReader.open(dir)) {
        IndexSearcher searcher = new IndexSearcher(reader);
        Query query =
            new BooleanQuery.Builder()
                .add(new TermQuery(new Term("f", "c")), Occur.SHOULD)
                .add(new PhraseQuery("f", "a", "b"), Occur.SHOULD)
                .build();
        Weight weight = searcher.createWeight(searcher.rewrite(query), ScoreMode.COMPLETE, 1f);
        LeafReaderContext leaf = reader.leaves().get(0);
        Scorer reference = weight.scorer(leaf);
        assertEquals(0, reference.iterator().nextDoc());
        Scorer scorer = weight.scorer(leaf);
        assertEquals(0, scorer.iterator().nextDoc());
        scorer.twoPhaseIterator().approximation().docIDRunEnd();
        // expected: same score as without docIDRunEnd() (only f:c matches doc 0); actual: the non-matching phrase is added
        assertEquals(reference.score(), scorer.score(), 0f);
      }
    }
  }
}

Command

cp $REPRO/TestDisjunctionApproximationDocIdRunEnd.java $LUCENE/lucene/core/src/test/org/apache/lucene/search/
cd $LUCENE && ./gradlew :lucene:core:test --tests org.apache.lucene.search.TestDisjunctionApproximationDocIdRunEnd -Ptests.seed=DEADBEEF
# and with -Ptests.asserts=false: the same failure

Expected: score() on doc 0 gives 0.06880058, the same as a fresh scorer (only f:c matches doc 0).
Observed: expected:<0.06880058> but was:<0.59193045>, with assertions on and off.

All files of this repro: repro/LFV-1143/

More files, environment, full output350 words

TestDisjunctionApproximationDocIdRunEnd (one method, extends LuceneTestCase, package org.apache.lucene.search of the lucene/core tests). It compares score() on doc 0 of a scorer that made the docIDRunEnd() call with a fresh scorer's score() on the same document.

Keep the plain IndexSearcher: newSearcher() wraps the scorer in AssertingScorer, whose approximation does not forward docIDRunEnd(), so the test could not see the bug. The comment line is 123 columns, so checkGoogleJavaFormat flags it; its log shows the re-wrap ./gradlew tidy would apply (formatting only; tidy itself was not run). Use this method as the regression test; where it goes is upstream's choice (it was run only as its own class; TestDisjunctionDISIApproximation builds approximations directly rather than through a scorer). LuceneTestCase picks the codec per seed, but the numbers did not change: this test gave the same values with seeds DEADBEEF (gradle), 5EED1143 and 5EED1144 (main jars) and 5EED1145 (10.2.0 and 10.6.0 jars), and the hunt's 3-document variant with DEADBEEF, 5EEDD7A1 and 5EEDD7A2. The hang case (the phrase also matches doc 0) is shown by a separate probe with a 10 s watchdog, not by this test.

Environment

lucene	`main 5adff78bb168 (12.0.0-SNAPSHOT): fails; 10.2.0, 10.4.0, 10.5.0, 10.5.2, 10.6.0 (run): fail; 10.1.0, 9.12.3: no docIDRunEnd() API (does not compile)`
jdk	`OpenJDK 25.0.3 (build 25.0.3+9-LTS)`
os	`Linux aarch64 (nothing platform-specific)`

Command (environment)

cp $REPRO/TestDisjunctionApproximationDocIdRunEnd.java $LUCENE/lucene/core/src/test/org/apache/lucene/search/
cd $LUCENE && ./gradlew :lucene:core:test --tests org.apache.lucene.search.TestDisjunctionApproximationDocIdRunEnd -Ptests.seed=DEADBEEF

Observed output

# gradle, scratch worktree at 5adff78bb168, -Ptests.seed=DEADBEEF -Ptests.asserts=false
TestDisjunctionApproximationDocIdRunEnd > testDocIdRunEndDoesNotChangeScore FAILED
    java.lang.AssertionError: expected:<0.06880058> but was:<0.59193045>
        at org.apache.lucene.search.TestDisjunctionApproximationDocIdRunEnd.testDocIdRunEndDoesNotChangeScore(TestDisjunctionApproximationDocIdRunEnd.java:59)
:lucene:core:test (FAILURE): 1 test, 1  failure

# same with assertions on: the same AssertionError and values

# with fix A or fix B: :lucene:core:test (SUCCESS): 1 test, assertions on and off

# supplementary, the hunt's probe with doc 0 = "a b c" (the phrase matches), -da
PROBE phrase matches doc 0 {'a b c','c','c'}: ... expected=0.71753067 score() after docIDRunEnd -> NO RETURN after 10 s, thread RUNNABLE at ...
PROBE swapped costs {'a b c','a b','a b'}: ... expected=0.50786304 score() after docIDRunEnd -> 0.50786304
# with fix B (assertions on and off): the first line returns 0.71753067

Why it looks unintended

docIDRunEnd() is documented not to move the iterator ("The current position of this iterator is not affected"); score() must give "the score of the current document".
The repro calls matches() once and never moves the approximation; Lucene itself calls docIDRunEnd() on positioned iterators.
Heap order decides: correct with the costs swapped, wrong otherwise, a hang if the phrase matches.

The full argument202 words

Contract. DocIdSetIterator.docIDRunEnd(): "The current position of this iterator is not affected by calling docIDRunEnd()" (DocIdSetIterator.java:245-263, item 3 at :252). Scorable.score(): "Returns the score of the current document matching the query" (Scorable.java:27); getChildren(): "Returns child sub-scorers positioned on the current document" (:57). Nothing in Scorer, TwoPhaseIterator or DisjunctionScorer says docIDRunEnd() invalidates matches().
Protocol followed. matches() is called once (through iterator().nextDoc()), as TwoPhaseIterator requires (TwoPhaseIterator.java:96-102), and the approximation never moves, so it stays in step with the iterator (Scorer.java:49-51).
Lucene itself asks positioned iterators for docIDRunEnd() and keeps using them. DisjunctionDISIApproximation.docIDRunEnd() calls docIDRunEnd() on the lead-heap clauses positioned on the current doc (DisjunctionDISIApproximation.java:227-230), and DenseConjunctionBulkScorer calls it on plain clauses' approximations (:49).
History. The verified-match list via next dates from LUCENE-6815 (1967050202d, 2015) and was harmless alone. #14363 (jpountz, 2025) added the topList()-based override ("takes the max #docIDRunEnd() across top clauses"); #16142 (costin, 2026) narrowed it to leadIterators.topList(). Neither PR or its reviews mention the shared links (#14363 has one empty review and no comments; #16142's comment, review comments and reviews discuss tests, run length, cost and CHANGES only).

The devil's-advocate check155 words

The strongest case for "not a bug": docIDRunEnd() literally promises only an unchanged position; DisjunctionDISIApproximation is @lucene.internal and TwoPhaseIterator @lucene.experimental; for two-phase iterators the run-end API is TwoPhaseIterator.docIDRunEnd(), which Lucene's own consumers use; and no in-tree caller makes the call.

Why it fails: the caller never names the internal class, it gets a plain DocIdSetIterator from public methods; the two methods answer different questions (the TwoPhaseIterator default means "no known run", TwoPhaseIterator.java:120-132); Lucene calls the approximation's docIDRunEnd() itself; "experimental" means the API may change, not that it may silently corrupt state. What survives is reach, hence priority Low. The closest precedent, #15324 / #15380, was fixed in the caller because MaxScoreBulkScorer advanced the approximation without matches(), which does break the protocol; here nothing moves. The contract does not promise that docIDRunEnd() has no other effect, so documenting the restriction or a caller-side answer (fix option D) would be a legitimate resolution.

Suggested fix

Recommended Walk the heap's top entries without writing next (fix B)

Walk the clause heap's top entries without topList(): a new package-private DisiPriorityQueue.maxDocIDRunEndOnTop(int) visits the same entries but never writes DisiWrapper.next (+40/-3 in 4 files, same cost as today). The repro passes with assertions on and off, the hang probe returns, and 156 tests in 13 classes (12 related suites and the repro) pass.

// DisjunctionDISIApproximation.docIDRunEnd(): no topList(), which rewrites DisiWrapper.next
if (leadTop.doc == doc) {
  maxDocIDRunEnd = leadIterators.maxDocIDRunEndOnTop(maxDocIDRunEnd);
}
// DisiPriorityQueueN (DisiPriorityQueue2 checks top and top2 the same way)
private int maxDocIDRunEndOnTop(int runEnd, int doc, int i) throws IOException {
  if (i < size && heap[i].doc == doc) {
    runEnd = Math.max(runEnd, heap[i].approximation.docIDRunEnd());
    final int left = leftNode(i);
    runEnd = maxDocIDRunEndOnTop(runEnd, doc, left);
    runEnd = maxDocIDRunEndOnTop(runEnd, doc, rightNode(left));
  }
  return runEnd;
}

All options and trade-offs4 options · 526 words

Options B and A were built and checked through gradle on 5adff78bb168; C and D were not built. The choice is upstream's.

Walk the heap's top entries without writing next (fix B) (validated by the checks) (recommended)

Add a package-private DisiPriorityQueue.maxDocIDRunEndOnTop(int), implemented in DisiPriorityQueue2 (top, and top2 if on the same doc) and DisiPriorityQueueN (the same recursion as topList()), that returns the max of the entries' approximation docIDRunEnd() without touching DisiWrapper.next, and call it from DisjunctionDISIApproximation.docIDRunEnd().

// DisjunctionDISIApproximation.docIDRunEnd(): no topList(), which rewrites DisiWrapper.next
if (leadTop.doc == doc) {
  maxDocIDRunEnd = leadIterators.maxDocIDRunEndOnTop(maxDocIDRunEnd);
}
// DisiPriorityQueueN (DisiPriorityQueue2 checks top and top2 the same way)
private int maxDocIDRunEndOnTop(int runEnd, int doc, int i) throws IOException {
  if (i < size && heap[i].doc == doc) {
    runEnd = Math.max(runEnd, heap[i].approximation.docIDRunEnd());
    final int left = leftNode(i);
    runEnd = maxDocIDRunEndOnTop(runEnd, doc, left);
    runEnd = maxDocIDRunEndOnTop(runEnd, doc, rightNode(left));
  }
  return runEnd;
}

Trade-offs

Same cost as today: only the entries on the current doc are visited. The price is a new method on a sealed internal class and a second heap walk to keep in step with topList(). No benchmark was run.

What the checks validated

Gradle in a scratch worktree at 5adff78bb168, JDK 25, seed DEADBEEF: the repro passes with assertions on and off; the hunt's hang probe returns 0.71753067 instead of hanging (on and off).
TestDisjunctionDISIApproximation (5), TestDisiPriorityQueue (2), TestBooleanQuery (30), TestBoolean2 (10), TestBooleanOr (7), TestBooleanMinShouldMatch (18), TestBooleanScorer (20), TestBooleanQueryVisitSubscorers (7), TestDisjunctionMaxQuery (23), TestDisjunctionScoreBlockBoundaryPropagator (1), TestDenseConjunctionBulkScorer (26), TestTwoPhaseIterator (6) and the repro: 156 tests, 0 failures.
ecjLintMain passes; checkGoogleJavaFormat flags only the repro's comment line.

Iterate the whole lead heap and keep the entries on the current doc (fix A, the hunt's sketch) (validated by the checks)

In DisjunctionDISIApproximation.docIDRunEnd(), replace the topList() loop with for (DisiWrapper w : leadIterators) and if (w.doc == doc). +6/-2 in one file.

for (DisiWrapper w : leadIterators) {
  if (w.doc == doc) {
    maxDocIDRunEnd = Math.max(maxDocIDRunEnd, w.approximation.docIDRunEnd());
  }
}

Trade-offs

The smallest change, but O(lead heap size) per call instead of O(lead clauses on the current doc), plus an iterator allocation; the method's comment (DisjunctionDISIApproximation.java:222-225) and #16142 show that this per-call cost was a deliberate concern. Equal cost for a two-entry heap.

What the checks validated

Same setup: the repro passes with assertions on and off; the same 156 tests pass. Earlier hunt runs: the repro and a cross-model reviewer's test pass, and TestDisjunctionDISIApproximation passes.

Let DisjunctionScorer.TwoPhase keep its own links (not built)

Record the verified clauses in an array in matches() and link them in getSubMatches().

Trade-offs

Robust against any other walk of next between matches() and score(), but it adds work to the per-document scoring path of every two-phase disjunction.

Document the restriction (not built)

Say on Scorer#twoPhaseIterator() or in DisjunctionScorer that the approximation's docIDRunEnd() must not be called between matches() and score().

Trade-offs

No code change, and a legitimate answer for an expert, @lucene.experimental API; but it leaves a silent wrong score and a hang behind a call that the DocIdSetIterator contract documents as not affecting the iterator's position, and that nothing documents as invalidating matches().

Downstream note

A downstream engine whose own bulk scorer or collector added this run-length call would give ordinary phrase-plus-term OR queries wrong scores, or a search thread that never finishes.

How it was found

A proof-obligation audit of DisjunctionScorer's contract: score() and getChildren() must reflect exactly the clauses that match the current document after any call the API allows between matches() and scoring. docIDRunEnd() on the approximation is such a call and rewrites the links that list is built from; a public-API JUnit test then witnessed the wrong score.

The method in full149 words

Method: semi-formal (a proof-obligation audit of a class's contract, witnessed by an executable repro).

The finder read DisjunctionScorer.TwoPhase against its obligation (the list of verified matches on the current doc stays valid until score() / getChildren()) and listed what may run in between under the documented API. docIDRunEnd() on the approximation is documented as position-neutral, but its override walks topList(), which writes the same next fields.
The finder then wrote a public-API JUnit test that fails on main with assertions on and off and on 10.6.0 and 10.2.0, a probe that shows the hang and the heap-order dependence, and the fix sketch (fix A).
An independent reproducer wrote its own test from the claim before reading the finder's; cross-model reviews, a devil's-advocate verifier and an adjudication reproduced it; a final repro check trimmed the test to the two-document repro on this page ("Prior reports, history and checks").

Prior reports, history and checks

Prior reports

No prior report in apache/lucene issues, PRs and commits (including the JIRA archive), the dev@, java-user@, issues@ and commits@ lists, downstream trackers or the web (searched 2026-11-02, re-checked 2026-11-03). Related: #15324 / #15380 (the same list, a different trigger and fix site).

Searches3 searches

No matching public report was found in the documented searches. Nearest items, none a report of this bug: #15324 / #15380 (the same verified-match list went out of sync because MaxScoreBulkScorer advanced the approximation without matches(); fixed in the caller, 2025-10-30); #14363 and #16142 (introduced and kept the relinking walk; their reviews discuss cost and run length only); 6f3c1d7ad69, #16307, #16450 and #16512 (other docIDRunEnd work on two-phase iterators). Limits: keyword search; private reports are not visible.

What was searched, when, and what turned up
Where	When	How	Found
GitHub issues, PRs and commits (apache/lucene incl. the migrated JIRA archive)	2026-11-02, re-checked 2026-11-03	a scripted prior-report search (16 issue/PR, 3 commit and 4 JIRA queries), an earlier 23-query run and 20 manual searches (for example verifiedMatches, getSubMatches, DisiWrapper next, DisjunctionSumScorer hang, approximation docIDRunEnd side effect); every issue and PR updated since 2026-10-31; on 2026-11-03 no issue or PR on docIDRunEnd, DisjunctionDISIApproximation, verifiedMatches, topList or DisjunctionScorer updated since 2026-11-02	no report; #15324 / #15380 related
Open PRs and recent commits	2026-11-02, re-checked 2026-11-03	the changed files of all 409 open PRs; upstream main commits by path since 2026-06-01	none touches the fix site; no commit to the four files since 2026-10-01
Mailing lists, downstream, web	2026-11-02	20 list queries (dev@, java-user@, issues@, commits@), Elasticsearch, OpenSearch and Solr trackers, 6 web queries; 339 distinct hits in the scripted search, none with high overlap	only PR and commit mirrors and unrelated threads

History

Introduced in 10.2.0 by #14363 (9792d88041e, 2025-03-17), which added docIDRunEnd() to the disjunction approximation with a topList() walk; #16142 (10.5.0) narrowed the walk but kept topList(). Fails on every release that was run (10.2.0, 10.4.0, 10.5.0, 10.5.2, 10.6.0) and on main; 10.1.0 has no docIDRunEnd() API. Not fixed upstream.

Versions tested6 rows

Introduced by #14363 "Implement #docIDRunEnd() on DisjunctionDISIApproximation." (jpountz): 9792d88041e on main and 0e6911da5de on branch_10x, author date 2025-03-17. Swapping the pre-#14363 class (from 0e6911da5de^) into the 10.2.0 jars makes the repro pass, which isolates the commit. Kept by #16142 "Optimize disjunction docIDRunEnd" (costin; 8e90c661ca6 / 023481ae9b1, merged 2026-06-02, first in 10.5.0), which narrowed the walk to leadIterators.topList(). The other half, the verified-match list built through next, dates from LUCENE-6815 (1967050202d, 2015). DocIdSetIterator#docIDRunEnd() itself is new in 10.2.0 (#14312), so no release had the API without the bug. Not fixed upstream: upstream main 1c2072f0284 (2026-11-02) is 3 commits past 5adff78bb168, none in lucene/core's search package (one, #16964, edits CheckIndex.java in the index package).

Versions and branches executed, with results
Version or commit	Verdict	Result
main `5adff78bb168`	affected	gradle: fails with assertions on and off; passes with fix A or fix B
10.6.0, 10.5.0, 10.4.0	affected	the 3-document variant: `expected:<0.045729928> but was:<0.71753067>` (-da)
10.5.2	affected	the independent reproducer's test: the same numbers (-da)
10.2.0	affected	fails with -ea and -da (also this two-document test, -da, in the fact check); first release with the bug
10.2.0 jars + pre-#14363 class	bisect	passes: isolates #14363
10.1.0, 9.12.3	no API	no `DocIdSetIterator#docIDRunEnd()`: the repro does not compile

Independent checks

A finder, an independent reproducer, cross-model reviews (OpenAI GPT-5.6), a devil's-advocate verifier, an adjudication, an audit and a final repro check reproduced it and agree on the root cause. This doc's gradle runs reproduced it on main and validated two fixes.

Checks and verdicts366 words

The refute-lens review ran on main ea57360912ed, where the code on this path is the same as at 5adff78bb168; its public test re-run on 5adff78bb168 fails with assertions on and off. Every other check ran on 5adff78bb168.

Each independent check and its verdict
Check	Verdict	What it found
finder	reproduced	3-document test fails on main (on and off), 10.6.0 and 10.2.0; probe shows the hang and the heap-order dependence; fix A passes the test and `TestDisjunctionDISIApproximation`.
cross-model review (OpenAI GPT-5.6)	real	Re-ran the repro and the probe.
independent reproducer	real	Own test fails on main (on and off), 10.6.0, 10.5.2 and 10.2.0; probe: an extra child in `getChildren()`, a correct control and swapped-cost case, and the hang.
cross-model review, refute lens (OpenAI GPT-5.6, executed)	real (high)	Own public test (1.7402484 instead of 1.0) fails over three seeds while its sequencing control passes; fix A fixes it; the "position only" counter-argument rated moderate and answered.
devil's-advocate verifier	real (high)	Rebutted the narrow-contract, misuse, protocol and intent readings; only reach is left.
adjudication	confirmed, Low	Re-ran the reproducer's test on clean main (on and off); the regression test must use a plain `IndexSearcher`.
separate re-run, audit and final repro check	confirmed	The tests fail on main (on and off) and pass with fix A; the hang re-observed; the audit classed it as silent wrong results; the final check set the class to minor because no Lucene code path makes the call.
this doc	reproduced; fixes A and B validated	Gradle without a fix fails (on and off); with each fix the repro and 156 tests pass; fix B also ends the hang.
independent fact check	real as described	Re-ran the gradle repro (on and off, fixes A and B), the main, 10.2.0 and 10.6.0 jars over five seeds, and a probe (the extra child on the repro's documents; a three-clause case); its wording findings are fixed in this doc.
cross-model fact check (OpenAI GPT-5.6)	real	Without running code: verified the mechanism, line ranges, runs and fixes; rated the intended-behaviour case moderate; its contract and trigger wording findings are fixed in this doc.

Diagram file

The diagram as a still image, generated with AI and checked against the code. It is under 10 MB, the limit for a GitHub attachment.

PNG 494 KB

Sources and provenance

Finding record, Lucene and upstream sources, repro files and diagram notes1,408 words

Finding record

LFV-1143, snapshot of 2026-11-06

Registry title: Calling docIDRunEnd() on a two-phase disjunction's approximation between matches() and score() corrupts DisjunctionScorer's list of verified matches, so score() and getChildren() include a clause that does not match the current document, or never return (since 10.2.0)
Area: core / search (DisjunctionScorer two-phase verified matches vs DisjunctionDISIApproximation.docIDRunEnd)
Priority: Low, rank 603: Calling docIDRunEnd() on a disjunction scorer's approximation between matches() and score() gives a wrong score, or a score() call that never returns, in every release since 10.2.0. Nothing in Lucene makes that call, so only contrived custom scorer code hits it.
Impact class: minor
Impact note: Only application code that drives scorers itself (a custom BulkScorer, collector or scorer composition) and calls docIDRunEnd() on twoPhaseIterator().approximation() of a positioned disjunction scorer (for example BooleanQuery SHOULD term + SHOULD phrase or doc-values range clause, scored with COMPLETE) after matches() and before score()/getChildren() is hit; then, when the two-phase clause sits on top of the lead heap, score() silently adds that non-matching clause and getChildren() reports it, and when the two-phase clause also matches, score()/getChildren() loop forever (-ea and -da alike); default codec, IndexWriterConfig and a plain IndexSearcher suffice and no scale is needed, but no Lucene code makes this call, and tests run through newSearcher()/AssertingScorer cannot see it because its approximation wrapper does not forward docIDRunEnd() [exec + read].
Summary: You index two documents {0: "b a c", 1: "c"} and build BooleanQuery(SHOULD f:c, SHOULD phrase "a b") with ScoreMode.COMPLETE; you move a scorer to doc 0 with iterator().nextDoc() and ask twoPhaseIterator().approximation().docIDRunEnd() (it returns 1 and the scorer stays on doc 0). You expect score() to return 0.06880058, the f:c score a fresh scorer gives, because docIDRunEnd() is documented not to affect the iterator; you get 0.59193045 with no error, because the phrase, which does not match doc 0, is added in. docIDRunEnd() walks the lead heap's topList(), which rewrites the same DisiWrapper.next links that matches() used to build the list of verified matches, and if the phrase also matches doc 0 the list becomes a cycle and score() never returns.
Introduced: 9792d88041ed61d0ce3a9a6c58a0c11ce0202a41 on main and its branch_10x cherry-pick 0e6911da5de89c21146beaef7149f01550f248aa, #14363 'Implement #docIDRunEnd() on DisjunctionDISIApproximation.' (jpountz): its docIDRunEnd() takes the maximum run end over approximation.topList(), and DisiPriorityQueue2.topList() rewrites DisiWrapper.next [exec: shadowing the pre-#14363 DisjunctionDISIApproximation (0e6911da5de^ = 53cef55e562) into the 10.2.0 jars turns the repro from fail into pass]. #16142 (8e90c661ca6 on main, 023481ae9b1 on branch_10x, 'Optimize disjunction docIDRunEnd', first in 10.5.0) narrowed the walk to leadIterators.topList(), which still relinks, so the bug stayed. The other half of the aliasing, DisjunctionScorer.TwoPhase building its verified-match list through DisiWrapper.next, dates from 1967050202d (LUCENE-6815, 2015) and was harmless alone [read]. (2025-03-17 (author date 2025-03-17T22:28:18+01:00; PR merged 2025-03-17T21:28:19Z; cherry-picked to branch_10x the same day))
First affected version: 10.2.0 [exec: the repro fails on 10.2.0 with -ea and -da; 10.1.0 and 9.12.3 have no DocIdSetIterator.docIDRunEnd() (added by #14312, fe913e54d9e, also first released in 10.2.0), so the repro does not compile there and no release had docIDRunEnd() without the bug]
Affected versions: every release from 10.2.0 through 10.6.0 (released 2026-10-29, 4ae946e5a9b) and main 5adff78bb168 (12.0.0-SNAPSHOT). Executed with the finder's one-test JUnit: main (-ea and -da; finder, the intended-behaviour check, history and re-runs), 10.6.0 (-da; finder, the intended-behaviour check, history), 10.5.0 and 10.4.0 (-da, history), 10.2.0 (-ea and -da); with the independent reproduction's own test: main (-ea and -da; the independent reproduction and the adjudication), 10.6.0, 10.5.2 and 10.2.0 (-da) [exec]; 10.2.1, 10.2.2, 10.3.x and 10.5.1 by read (same loop, tag containment). Not affected: 10.1.0 and 9.12.3 (no docIDRunEnd API: compile error) [exec]. Not new in 10.6. Not fixed upstream: since 61e22dd3e60 (#16394, author date 2026-07-28, unrelated) no upstream main commit touches DisjunctionDISIApproximation, and DisjunctionScorer, DisiPriorityQueue2 and DisiWrapper have no commit since 2026-06-01 (author dates); main is 3 commits past 5adff78bb16, none in lucene/core's search package (one, #16964, edits CheckIndex.java in the index package) [read]. Executed on Linux aarch64 with JDK 25.0.3 only.
Status: documented (brief, checks and a per-finding doc exist)
Verification: Confirmed by AI-run checks: an independent repro, a cross-model review that agreed, an intended-behaviour check, a prior-report search and a skeptical audit; not yet triaged by a person.
Found by: semi-formal: proof-obligation audit
Upstream and prior reports: none found (2026-11-02, on top of the finder's, the independent reproduction's, the intended-behaviour check's and the adjudication's searches; rechecked the same day): a scripted prior-report search over 51 queries (16 GitHub issue and PR, 3 commit, 2 downstream on Elasticsearch, OpenSearch and Solr, 20 mailing-list over dev@, java-user@, issues@ and commits@, 4 JIRA-archive and 6 web; 339 distinct hits, none with high overlap) plus the finder's 23-query run (80 hits, none highly relevant), 20 manual GitHub searches (incl. open-PR, downstream and phrase variants), the file lists of all 409 open apache/lucene PRs (only #11928, #12194, #14371, #101, #113 and #463 touch these files, all from 2021-2025 and unrelated) and of the 34 PRs updated since 2026-10-31 (none touches them), upstream main commits by path since 2026-06-01 with author dates, and CHANGES.txt on main (no entry): no report of docIDRunEnd() on a two-phase disjunction's approximation relinking DisiWrapper.next and corrupting DisjunctionScorer's verified-match list. Related, not the same bug: #15324 / #15380 (closed / merged 2025-10-30: the same verified-match list went out of sync because MaxScoreBulkScorer advanced a disjunction's approximation without calling matches(); fixed in the caller, and the added assertVerifiedMatchesApproximationDocID does not end on a cycle); #14363 and #16142 (introduced and kept the relinking topList() walk; their reviews discuss only cost and run length); 6f3c1d7ad69, #16307, #16450 and #16512 (other docIDRunEnd work on two-phase iterators). Same-day recheck: GitHub issues and PRs mentioning docIDRunEnd, DisjunctionDISIApproximation, DisjunctionScorer, topList or verifiedMatches updated since 2026-10-31: 0 each; the apache/lucene issues and PRs updated since 2026-10-31 (the last 48 h) include none on this defect.
Lucene commit tested: main 5adff78bb168 (2026-11-02) through gradle in a scratch worktree with assertions on and off, without a fix and with fixes A and B (156 tests in 13 classes: 12 related suites and the repro; ecjLintMain and checkGoogleJavaFormat for B; and the hunt's hang probe), and through a standalone JUnit runner on the main jars. An independent fact check re-ran the gradle repro (on and off, fixes A and B), the standalone runner on the main, 10.2.0 and 10.6.0 jars (five seeds in all, the same values) and a probe on the repro's own documents. The hunt ran this test or its 3-document variant on 10.1.0 and 9.12.3 (no docIDRunEnd() API: does not compile), 10.2.0, 10.4.0, 10.5.0, 10.5.2, 10.6.0 and main. Only Linux aarch64 with OpenJDK 25.0.3 was run.

Lucene and upstream sources

Finding record: lucene/core/src/java/org/apache/lucene/search/DisjunctionDISIApproximation.java:220-233 (at 5adff78bb168) (docIDRunEnd: when leadTop.doc == doc it walks leadIterators.topList(), :228, and calls docIDRunEnd() on every sub-approximation, :229; fix site), :30 (@lucene.internal)
lucene/core/src/java/org/apache/lucene/search/DisiPriorityQueue2.java:55-66 (at 5adff78bb168) (topList: top.next = null, and top2.next = top when both are on the same doc)
lucene/core/src/java/org/apache/lucene/search/DisjunctionScorer.java:118-157 (at 5adff78bb168) (TwoPhase.matches builds verifiedMatches through w.next: plain clauses :123-139, first matching two-phase clause :150-151), :95-106 (getSubMatches prepends confirmed two-phase clauses with w.next = verifiedMatches, :99), :108-116 (assertVerifiedMatchesApproximationDocID walks the list), :178-193 (score() and getChildren() walk the list until null)
lucene/core/src/java/org/apache/lucene/search/DisjunctionSumScorer.java:40-46 (at 5adff78bb168) (score sums w.scorable.score() for w = topList until null; the -da spin at :43)
lucene/core/src/java/org/apache/lucene/search/DocIdSetIterator.java:245-263 (at 5adff78bb168) (docIDRunEnd contract: 'The current position of this iterator is not affected by calling docIDRunEnd()')
lucene/core/src/java/org/apache/lucene/search/TwoPhaseIterator.java:29 (at 5adff78bb168) (@lucene.experimental), :96-100 (matches() at most once), :118-132 (docIDRunEnd default returns approximation().docID(), 'no known matching run')
lucene/core/src/java/org/apache/lucene/search/DenseConjunctionBulkScorer.java:47-53 (at 5adff78bb168) (two-phase clauses use TwoPhaseIterator.docIDRunEnd(), so this in-tree path never reaches the bug); FilterDocIdSetIterator.java (does not override docIDRunEnd, so AssertingScorer's approximation wrapper hides the bug)
The finding in 30 seconds: lucene/core/src/java/org/apache/lucene/search/DisjunctionScorer.java:83-84, (at 5adff78bb168) 95-116, 118-157, 178-193
lucene/core/src/java/org/apache/lucene/search/DisjunctionSumScorer.java:40-46 (at 5adff78bb168)
lucene/core/src/java/org/apache/lucene/search/DisjunctionDISIApproximation.java:220-233 (at 5adff78bb168)
lucene/core/src/java/org/apache/lucene/search/DisiPriorityQueue2.java:55-66; (at 5adff78bb168) DisiPriorityQueueN.java:72-106; DisiWrapper.java:36
Who is affected: lucene/core/src/java/org/apache/lucene/search/DenseConjunctionBulkScorer.java:47-53; (at 5adff78bb168) TwoPhaseIterator.java:29, 130-132
lucene/test-framework/src/java/org/apache/lucene/tests/search/AssertingScorer.java:225-258; (at 5adff78bb168) lucene/core/src/java/org/apache/lucene/search/FilterDocIdSetIterator.java (at 5adff78bb168) (no docIDRunEnd override)
Why it looks unintended: lucene/core/src/java/org/apache/lucene/search/DocIdSetIterator.java:245-263; (at 5adff78bb168) Scorable.java:27, 56-60; Scorer.java:45-51; TwoPhaseIterator.java:29, 96-102, 112-132
lucene/core/src/java/org/apache/lucene/search/DisjunctionDISIApproximation.java:30, (at 5adff78bb168) 229; DenseConjunctionBulkScorer.java:47-53
Suggested fix: lucene/core/src/java/org/apache/lucene/search/DisjunctionDISIApproximation.java:220-233 (at 5adff78bb168) (the comment at :222-225)
Diagrams: lucene/core/src/java/org/apache/lucene/search/DisjunctionScorer.java:95-157; (at 5adff78bb168) DisjunctionDISIApproximation.java:220-233; DisiPriorityQueue2.java:55-66

Other statements trace to the hunt's working files (models, logs, reviews and briefs), which are not published.

Repro files

TestDisjunctionApproximationDocIdRunEnd.java (lucene/core, package org.apache.lucene.search, 1 test): repro/LFV-1143/TestDisjunctionApproximationDocIdRunEnd.java (63 lines)

About the diagrams

The diagram illustrates the repro's scenario (main 5adff78bb168, DisjunctionScorer.java:118-157, DisjunctionDISIApproximation.java:220-233, DisiPriorityQueue2.java:55-66).

assets/LFV-1143/bug.png (494 KB)

Built 2026-11-06 from the hunt's record of this finding and its doc file. Lucene main under test in this hunt: ea57360912ed. Every listed finding was re-run on Lucene main e1faeba08bde on 2026-11-02.

The finding in 30 seconds

Background: two-phase disjunctions and docIDRunEnd()

The finding, step by step

Who is affected

The mechanism (main 5adff78bb168, lucene/core/src/java/org/apache/lucene/search/)

Minimal repro

Environment

Observed output

Why it looks unintended

Suggested fix

Walk the heap's top entries without writing next (fix B) (validated by the checks) (recommended)

Iterate the whole lead heap and keep the entries on the current doc (fix A, the hunt's sketch) (validated by the checks)

Let DisjunctionScorer.TwoPhase keep its own links (not built)

Document the restriction (not built)

Downstream note

How it was found

Prior reports, history and checks

Prior reports

History

Independent checks

Diagram file

Sources and provenance

Finding record

Lucene and upstream sources

Repro files

About the diagrams

The mechanism (main `5adff78bb168`, `lucene/core/src/java/org/apache/lucene/search/`)