Luke: 'next' after the last full page of a search with more than 1,000 hits shows an empty page and disables 'previous'

Luke's Search tab pages through results 10 at a time. Unless 'exact hits count' is ticked, Lucene counts hits exactly only up to 1,000, so a bigger result reports a lower-bound total such as '1010+ hits', and Luke keeps 'next' enabled because it cannot tell from the total where the results end. On the last page of a result whose last page is full (more than 1,000 hits, a multiple of 10), one more 'next' click calls SearchImpl.nextPage(), which first adds 1 to its page counter, skips its end-of-results check (that check runs only for exact totals), searches after the last hit, finds nothing and returns the 0-hit result as a page. SearchResults.of sets a page's offset only inside its loop over the hits, so that page reports offset 0. The tab shows it: an empty table, the range '1 ~ 0', 'previous' disabled. Nothing in the index changes and the results shown before were right, but the user has to re-run the search. The javadoc of Search.nextPage() promises an empty result when there are no more results. The code came in with the Luke module in 8.1.0 and every release run since has it; a fix of about 20 lines in two files moves the page counter only to a page that exists and sets the offset outside the loop.

1. SearchImpl.nextPage() adds 1 to its page counter first: currentPage += 1; (SearchImpl.java:342).
2. Its end check, totalHits.value() == 0 || (relation == EQUAL_TO && currentPage * pageSize >= value) (:344-349), fires only when the total is exact.
3. Past the last remembered hit it runs a new search and returns Optional.of(search()) (:360), also when that search found 0 hits. search() builds the page with offset currentPage * pageSize (:331-332).
4. SearchResults.of sets res.offset = offset; inside for (ScoreDoc sd : docs) (SearchResults.java:69-76, the assignment at :75), and the field starts at 0 (:40). A page with no hits therefore reports offset 0.
5. The tab shows every page it gets: searchModel.nextPage().ifPresent(this::populateResults); (SearchPanelProvider.java:546-548). populateResults (:614-623) shows the range offset + 1 ~ offset + size, enables 'previous' only when offset > 0, and keeps 'next' enabled for a lower-bound total.

Without a sort and with 'exact hits count' off, Luke collects with a total-hits threshold of 1,000 (SearchImpl.java:74, :316-318). The collector reports a lower bound once it has counted more than 1,000 hits and its 10 slots are full (TopScoreDocCollector.java:154-163). A partial last page never fills them, so it comes back with an exact total ('1005 hits') and the tab disables 'next'. A full last page comes back as '1010+ hits', and the next click asks for a page that does not exist.

The empty page's own search counted every hit, so its total is exact ('1010 hits'). Later 'next' clicks return empty and do nothing; 'previous' stays disabled. For sorted searches the total stays a lower bound because of the companion bug LFV-1036, and every 'next' returns another empty page.

Same root, model API only (the tab's buttons never reach these): prevPage() also moves before it checks (SearchImpl.java:375), so prev after an empty next returns the last page again, next or prev after a prev on the first page throws LukeException ('Search session not started.'), and, when the last page is partial (e.g. 1 hit, page size 10), prev after two nexts past the end throws NegativeArraySizeException; with a full last page, as for 1,010 hits, that prev returns another 0-hit page.

• Who is hit. Users of Luke (lucene/luke, started with bin/luke.sh or bin/luke.cmd from the binary distribution) who page to the very end of a search.
• Default vs opt-in. The defaults: the page size is fixed at 10 (SearchPanelProvider.java:98), no sort, and 'exact hits count' unticked (a plain checkbox, :142). The 'More Like This' search passes no sort and exactHitsCount false (:578-579), so it takes the same path [read]. Ticking 'exact hits count' avoids it for unsorted searches. Sorted searches over 1,000 hits strand for any hit count, through the companion LFV-1036 (the sorted path ignores exactHitsCount).
• Scale. More than 1,000 hits, a full last page, and 101 or more 'next' clicks.
• Effect. The tab loses its place: an empty table, '1 ~ 0', 'previous' disabled; re-running the search recovers. No exception in the GUI path, no wrong hits, no index change. Assertions on or off make no difference.
• Likelihood. Rare: paging by hand to the end of more than 1,000 hits is unusual. Priority Low.

The test uses the public SearchFactory and Search API; it sits in Luke's package only because TestSearchImpl does too. 1,010 is the smallest hit count past 1,000 with a full last page at page size 10. It passes null as the fields to load (all stored fields; the tab passes the fields ticked on its Field Values sub-tab); the documents store no fields, so this does not change the result. The file uses the deprecated MatchAllDocsQuery constructor so it also runs on older releases; on main, MatchAllDocsQuery.INSTANCE replaces it. As the regression test: this same test, for example as a method next to testNextPageNoMoreResults in TestSearchImpl.

Not shown by the test (the tab's button logic replayed on the model shows it; the Swing tab itself was not driven): the '1 ~ 0' range and the disabled 'previous' button.

cp TestNextPagePastLastPage.java $LUCENE/lucene/luke/src/test/org/apache/lucene/luke/models/search/
cd $LUCENE && ./gradlew :lucene:luke:test --tests TestNextPagePastLastPage -Ptests.seed=DEADBEEF -Ptests.asserts=true
# assertions off: -Ptests.asserts=false (same result)

# gradle, stock 645d4fe14a20, -Ptests.seed=DEADBEEF -Ptests.asserts=true (stack trimmed; asserts=false identical)
   >     java.lang.AssertionError: got a page: size=0 offset=0
   >         at __randomizedtesting.SeedInfo.seed([DEADBEEF:207F0A726039F0BB]:0)
   >         at junit@4.13.2/org.junit.Assert.fail(Assert.java:89)
   >         at junit@4.13.2/org.junit.Assert.assertTrue(Assert.java:42)
   >         at org.apache.lucene.luke.models.search.TestNextPagePastLastPage.testNextPageAfterLastPage(TestNextPagePastLastPage.java:46)
:lucene:luke:test (FAILURE): 1 test, 1  failure

# with the fix: :lucene:luke:test (SUCCESS): 1 test (assertions on and off)
# 10.6.0 release jars, assertions on and off: java.lang.AssertionError: got a page: size=0 offset=0

• Contract. Search.java:116-122: 'Returns the next page for the current query. @return search results, or empty if there are no more results'. SearchResults.java:49 and :86: the offset is 'the offset of the current page'.
• The exact branch and its test. The EQUAL_TO branch (SearchImpl.java:344-349) returns Optional.empty() after the last page; TestSearchImpl.testNextPageNoMoreResults (TestSearchImpl.java:335-341, 20 hits, page size 10) asserts that, with exactHitsCount=true. Every search(...) call in TestSearchImpl passes true.
• History. The standalone Luke checked totalHits == 0 || currentPage * pageSize >= totalHits before Lucene 8.0, so a call past the end returned empty. DmitryKey/luke#163 ('Upgrade lucene 8.0.0', 2019-03-21) switched to the EQUAL_TO-only check when Lucene 8.0 made totals a lower bound past 1,000 hits (LUCENE-8060) and kept 'next' enabled for such totals, without handling an empty next page; every test it updated passes exactHitsCount=true. This is consistent with an overlooked case; it does not prove intent either way, and the captured record of #163 (PR title and body, patch, commit) does not mention the empty page. The standalone luke-swing-7.7.0 passes the test; luke-swing-8.0.0 fails it.

The strongest case for 'intended' (built by the devil's advocate and the reviews from another model family, OpenAI GPT-5.6; rated moderate): Luke's module exports no package and its models package calls itself 'Models and internal APIs for Luke', so the Search javadoc binds only Luke; and with a lower-bound total the model can only learn that the results ended by searching once more, so an empty page could be read as that probe's honest answer.

Why it fails: the bug does not rest on the API. The Search tab itself, with its own arguments and button logic, ends on the empty page with 'previous' disabled, and the repro uses those paging arguments (it passes null fields to load, meaning all stored fields, where the tab passes its ticked fields; the documents store none, so it does not matter). The probe is legitimate, but once it finds 0 hits the model knows there are no more results and still returns the probe as a page, with a wrong offset. The fix keeps the probe internal. All checkers rated the bug real.

The finder's patch was applied to a scratch worktree of 645d4fe14a20 and run through gradle. Moving the offset assignment alone is not a fix.

In nextPage(), compute next = currentPage + 1 and use it for the end check and the cached path; on the search path, move, search, and if the result has no hits move back and return Optional.empty(). In prevPage(), return empty when currentPage <= 0 before decrementing. In SearchResults.of, move res.offset = offset; after the loop.

      } else {
-        return Optional.of(search());
+        int prev = currentPage;
+        currentPage = next;   // next = currentPage + 1, used by the end check above
+        SearchResults res = search();
+        if (res.size() == 0) {
+          currentPage = prev;   // no such page: stay where we are
+          log.warning("No more next search results are available.");
+          return Optional.empty();
+        }
+        return Optional.of(res);
       }

Trade-offs

The extra search at the end still runs (with a lower-bound total it is the only way to learn that the results ended); only its empty result stays internal, and it adds no hits to the cache. In the tab the extra click now leaves the last page on screen with 'next' still enabled, doing nothing, as for any empty answer; disabling 'next' then would be separate GUI polish (not run). The empty probe search still replaces the session's total with the exact one, so a later 'previous' shows '1010 hits' instead of '1010+ hits' (accurate and harmless). As on stock code, an IOException from the probe search leaves the page counter moved. Of the model-API-only problems, nextPage() staying put when it returns empty removes the NegativeArraySizeException and the repeated last page, and the prevPage() change removes the LukeException after prev on the first page; the offset change removes neither (executed by the independent fact check, one hunk at a time).

What the checks validated

• Gradle in a scratch worktree of 645d4fe14a20 (2026-11-01): the repro fails on stock with assertions on and off and passes with the fix, on and off; org.apache.lucene.luke.models.search.* (the repro and the 17 upstream TestSearchImpl tests) passes with seeds DEADBEEF and CAFEBABE; the whole :lucene:luke:test passes with seed DEADBEEF (12 classes, 98 tests, 0 failures; 17 tests skipped, all in TestDocumentsImpl per the JUnit XML reports of the independent fact check's re-run).
• Earlier, with the fixed classes shadowed in: the minimal test and the finder's, the independent reproducer's and the cross-model adversary's tests pass; the refute-lens cross-model review ran the 17 upstream TestSearchImpl tests with the fix, 0 failures.
• Exhaustive enumeration: with this fix only the companion's sorted exact-count mismatches remain; with both fixes, 0 of 524,640 call sequences and 0 of 37,952 click sequences diverge from the contract.

Looks like a one-liner.

Trade-offs

nextPage() still returns a 0-hit page. In the default unsorted case it now shows as '1011 ~ 1010' with 'previous' enabled and 'next' disabled (the probe's total is exact), so 'previous' gets back to page 101. For sorted searches over 1,000 hits, where the companion LFV-1036 keeps the total a lower bound, next, next, previous through the tab's own buttons then throws NegativeArraySizeException (exhaustive enumeration, variant C; re-run by the independent fact check).

Make the tab's nextPage or populateResults skip a page with no hits.

Trade-offs

Hides the symptom in the tab but leaves Search.nextPage() breaking its javadoc and the model-API exceptions in place. Not run.

Method: non-formal first signal, then a bounded-exhaustive enumeration. Earlier executed probes on the Luke search model (one through the model API, one clicking only the buttons the Search tab enables) showed the move-before-check pattern and sorted searches over 1,000 hits ending on an empty page with offset 0, but judged unsorted paging safe.

The hunt unit for Luke's search paging then enumerated the paging state machine on the stock build: every sequence of up to 6 calls over next, previous and search after an initial search, over 480 configurations (hit counts 0 to 2,000, page sizes 1 to 1,000, exact count on and off, three sorts, 1 or 3 segments), 524,640 sequences in all, compared after each call with a reference written from the javadoc; and every click sequence of up to 5 that uses only enabled buttons, compared with the tab's own button logic. It found the unsorted, default-settings trigger. Fixing this root alone removes every mismatch except the companion's; fixing both leaves 0 of 524,640 call sequences and 0 of 37,952 click sequences diverging. A deliberately wrong reference fails 229,548 and 15,532 sequences, so the check is not vacuous.