MMapDirectory puts a slice's read advice on the wrong bytes when the slice starts inside an mmap chunk and reaches or crosses that chunk's end (10.0.0+)

MMapDirectory maps a file in chunks (16 GiB by default on 64-bit JVMs, smaller when the application passes a maxChunkSize). With a read-advice function set, IndexInput#slice(name, offset, length, IOContext) gives the slice its own madvise hint, and the compound-file reader uses it so that each inner file advises its own pages. A slice that starts inside a chunk and reaches or crosses that chunk's end keeps the whole first chunk and remembers where it starts inside it, in MultiSegmentImpl.offset. slice(..., IOContext) then advises the slice from its offset 0, and advise() never adds that in-chunk start, so the hint starts at the chunk start instead of the slice start. #14109 fixed the same omission for prefetch() only. In the repro (1 MiB chunks) a slice of [512 KiB, 1.5 MiB) with a RANDOM hint puts MADV_RANDOM on [0, 1 MiB). In a compound file those bytes belong to other inner files, and a short slice straddling a chunk boundary can get none of its own pages advised. Every byte read and every query result stays correct: it is a performance defect, needing a non-NORMAL read advice (opt-in on 10.3.0+ and main, the default on 10.1-10.2.x) and a slice of at least one page that starts inside a chunk and reaches or crosses its end. Affected: every 10.x release and main. No slowdown was measured.

1. The slice keeps the whole first chunk. buildSlice (:653-700) copies the chunks the slice touches (:663-667), truncates the last one (:670) and keeps only the in-chunk start, offset = offset & chunkSizeMask (:672). A single-chunk slice cuts its segment to exactly the slice (slices[0].asSlice(offset, length), :680); a multi-chunk slice passes offset to new MultiSegmentImpl(...) (:688-698), which stores it (:848, :871).
2. The slice is advised from its offset 0. slice(String, long, long, IOContext) (:623-650) returns early for NORMAL advice (:627-630) and for slices shorter than a page (:632-640), then calls:

slice.advise(                                   // :642
    0,
    slice.length,
    segment -> {
      nativeAccess.madvise(segment, advice);
      return true;
    });

1. advise() does not know about the slice offset. It takes segments[(int) (offset >> chunkSizePower)] (:405) and offset &= chunkSizeMask (:406), clamps the length to that segment (:408-413), rounds to pages (:414-427) and advises segment.asSlice(offset, length) (:429-430).
2. MultiSegmentImpl translates every other data position. It adds offset in seek (:886-889), the positional read* methods (:896-919), segmentSliceOrNull (:921-934), buildSlice (:936-940) and prefetch (:942-946, the #14109 fix), and subtracts it in getFilePointer() (:891-894) and in the positions it reports in errors (:880-883). It has no advise override. A whole multi-chunk file and its clones have offset 0 (:93-105), so only slices that start inside a chunk are affected, together with their clones (which keep the offset, :578) and slices of them (buildSlice adds the parent's offset, :936-940). A slice that ends exactly on a chunk boundary also gets a 0-byte second segment (:665-670), so it is a MultiSegmentImpl too.

In general: slices shorter than a page are never advised (:632-640). A longer slice that starts at in-chunk offset o > 0 of chunk k and ends at or after (k+1)*C is advised on [k*C, k*C + min(length, C)), page-rounded, instead of its part of the first chunk, [k*C + o, (k+1)*C) (with the fix, a start that is not page-aligned is rounded down to its page). Single-chunk slices and slices that start on a chunk boundary (o = 0) are correct; they passed as controls wherever a verifier included them (the first independent reproducer both; the cross-model reviews a single-chunk control).

On main those other inner files ask for no RANDOM hint; in the default codec only stored fields (.fdt), term vectors and the kNN vector files do.

• Effect. MADV_RANDOM turns readahead off for the pages it covers, so sequential reads of the neighbouring inner files (for example a merge reading .tim or .pos) take more, smaller page faults. ADVISE_BY_CONTEXT returns SEQUENTIAL for MERGE and FLUSH contexts (MMapDirectory.java:159-161), which makes readahead more aggressive on pages the slice does not own. A short slice straddling a chunk boundary can get no hint on its own pages while its readAdvice field still says RANDOM, which isEfficientFor() trusts (MemorySegmentIndexInput.java:381-392) when the compound reader decides whether to reuse a mapping (Lucene90CompoundReader.java:196). None of this was measured.
• Correctness is unaffected. The hint is NORMAL, RANDOM or SEQUENTIAL on a read-only mapping of the same file. Reads go through the slice's own segments and offset, which the bug does not touch.
• Platform. madvise is used only on Linux and macOS (NativeAccess.java:41-46) and only on 64-bit JVMs (PosixNativeAccess.java:66-67).
• Default vs opt-in. On main and 10.3.0-10.6.0 the default read-advice function returns Optional.empty() (MMapDirectory.java:172-173), which maps to Constants.DEFAULT_READADVICE, NORMAL (MMapDirectory.java:326-328, Constants.java:186-189), and NORMAL advises nothing. It takes an explicit setting: in practice setReadAdvice, for example with ADVISE_BY_CONTEXT (MMapDirectory.java:157-170). The other setting, the system property org.apache.lucene.store.defaultReadAdvice (Constants.java:181-189), gives every file and context the same advice, so the whole compound file already carries it and the misplaced hint changes nothing visible (by reading). On 10.1.0-10.2.2 the default settings already advise compound inner files. On 10.0.0 only direct slice(..., ReadAdvice) calls are affected.
• Callers. The in-tree callers of slice(..., IOContext) are the compound readers: Lucene90CompoundReader (:195, :207, :210) and SimpleText's compound format (SimpleTextCompoundFormat.java:138-139, a debugging codec); applications and custom codecs may call it too.
• Layout. The slice must start inside a chunk and reach or cross that chunk's end. With the default 16 GiB chunk (MMapDirectory.java:185-186) that needs a compound file above 16 GiB, which the defaults make unusual: by default a merged segment is a compound file only up to 64 MB with byte-size merge policies such as the default TieredMergePolicy, or up to 65,536 documents with LogDocMergePolicy (CompoundFormat.java:38-42, :163-180). Realistic triggers are a smaller maxChunkSize (public constructor MMapDirectory(Path, long); the class javadoc recommends reducing the chunk size when mapping fails, MMapDirectory.java:49-50, though on 64-bit it says the chunk "should always be large (like 16 GiBytes)", :241-242) or raised compound-file thresholds together with segments above 16 GiB (the default TieredMergePolicy keeps merged segments below 5 GB, TieredMergePolicy.java:77, so that also takes a larger maximum segment size or a forceMerge). With 1 MiB chunks the end-to-end tests needed only 400 and 2,000 documents.
• Size of the misplaced range. At most one chunk (min(length, C)) of the same file. In the compound-reader path it happens once per inner-file open (segment open, merge), not per query; other callers of the public API choose their own frequency.

The test observes madvise through the per-mapping VmFlags in /proc/self/smaps (rr = random read), a practical way to see it from Java without changing Lucene, so it assumes Linux and MMapDirectory.supportsMadvise(); it ran (0 skipped) in every recorded run. The file is opened with IOContext.DEFAULT, which ADVISE_BY_CONTEXT maps to NORMAL, so nothing is advised on open and the only RANDOM hint is the slice's. The test asserts only that the hint does not land outside the slice, because read advice is best effort; a probe shows the fixed code still advises the slice's own part of chunk 0 ([524288, 1048576)), so the pass is not vacuous on the tested host. The test logic draws no random values (LuceneTestCase still randomizes locale, codec and other settings per seed); seeds DEADBEEF and 5EED1234 give the same failure. The same test is the suggested regression test (for example as a method of TestMMapDirectory). One gap as a regression test: it would also pass if smaps listed no matching mapping or no rr flag at all, and an unreadable /proc/self/smaps fails it instead of skipping it; a committer may want an extra check that the slice's own part carries rr (true with the fix, per the probe).

# Gradle, main 645d4fe14a20, stock (the same with -Ptests.asserts=false)
TestMultiChunkSliceReadAdvice > testRandomAdviceStaysInsideTheSlice FAILED
    java.lang.AssertionError: expected RANDOM advice only inside the slice [524288, 1572864), but file bytes [0, 1048576) got it
        at org.junit.Assert.assertTrue(Assert.java:42)
        at org.apache.lucene.store.TestMultiChunkSliceReadAdvice.testRandomAdviceStaysInsideTheSlice(TestMultiChunkSliceReadAdvice.java:47)
1 test completed, 1 failed

# with the fix: 1 test completed, 0 failed (assertions on and off, 0 skipped)
# probe of the rr-flagged file ranges: main [0, 1048576); with the fix [524288, 1048576)
# 10.6.0 and 10.5.2 release jars, -ea and -da: the same AssertionError

• The API exists to advise one inner file. The IndexInput#slice(String, long, long, IOContext) javadoc: "This is typically used by CompoundFormat implementations to modify the IOContext for specific files within the compound file" (IndexInput.java:126-139).
• The code says neighbours must not be advised. slice(..., IOContext) skips slices shorter than a page because advising them means "potentially overriding the advice of other files that share the same pages" (MemorySegmentIndexInput.java:633-638). The compound reader opens its extra mapping "without advice, like handle, so inner files advise their own pages" (Lucene90CompoundReader.java:202). In the #16883 review, jimczi wrote that "each inner file advises its own pages through its slice".
• The same omission was fixed as a bug. #14109, "prefetch may select the wrong memory segment for multi-segment slices", is under Bug Fixes for 10.2.0 (lucene/CHANGES.txt:1998-1999): "When sliced, a multi-segment index input uses an offset into the initial memory segment. This offset should be added to the prefetch offset to determine the absolute offset." uschindler's review there: "it's way too easy to add new methods but forget to look at code of methods ... prefetch just works". That fix overrode prefetch() only.
• No test pins the current range. TestMMapDirectory#testPrefetchWithMultiSegment "does not verify that the actual segment is prefetched" (TestMMapDirectory.java:468). TestMMapDirectory, TestMultiMMap and TestLucene90CompoundFormat pass on the stock jar and with the fix.

The strongest case for "intended": read advice is a best-effort hint. The javadoc says "only certain IOContext may be usable here" (IndexInput.java:130-131); advise() deliberately advises only the first chunk ("this case is rare enough that we don't try to optimize it and keep things simple", MemorySegmentIndexInput.java:409-411); the sub-page comment accepts that an inner file can override a neighbour's advice on shared pages; and the #16883 review deferred rounding the advised range inwards (jimczi: "We could, by rounding the advised range inwards so only pages fully inside the inner file get advised ... Could we do it in a follow up?").

Why it fails: each accepted imprecision is advice that does not reach far enough (later chunks) or one shared page at each end of an inner file. This defect moves the window up to a whole chunk before the slice, onto other files, and can miss the slice entirely; rounding inwards would act inside the same wrong window. The same missing offset in advise()'s other caller was treated as a bug. The works-as-designed reviewer (OpenAI GPT-5.6) rated the intended argument "moderate" and still judged the bug real; the devil's-advocate verifier found no argument that survives. Verdict: a defect, low severity.

One option was validated through Gradle in a scratch worktree of 645d4fe14a20; the study had validated the same diff by shadowing the class.

In MemorySegmentIndexInput.MultiSegmentImpl: advise() adds this.offset, and the prefetch() override keeps its bounds check but passes the slice-relative offset, because the base prefetch() calls advise() (:366), which now adds the offset once. Both callers of advise() then pass slice-relative offsets, the convention of every other MultiSegmentImpl method. +8/-1 lines in one file.

   @Override
   public boolean prefetch(long offset, long length) throws IOException {
     Objects.checkFromIndexSize(offset, length, this.length);
-    return super.prefetch(this.offset + offset, length);
+    return super.prefetch(offset, length);
+  }
+
+  @Override
+  boolean advise(long offset, long length, IOFunction<MemorySegment, Boolean> advice)
+      throws IOException {
+    // offsets are relative to this slice; segments[0] starts this.offset bytes before it
+    return super.advise(this.offset + offset, length, advice);
   }

Trade-offs

• prefetch() advises the same absolute range as before.
• Nuance: a multi-chunk slice whose start is not page-aligned now rounds its first page down (with chunks of at least a page, its segments[0] starts on a page boundary), so up to pageSize-1 bytes of the preceding inner file can still get the hint, while a single-chunk slice skips the shared first page (MemorySegmentIndexInput.java:414-427). That is the one-shared-page limitation the #16883 review accepted; the third option would make both consistent.
• Main only: on 10.x advise() is void advise(long, long, IOConsumer<MemorySegment>) and prefetch() returns void (10.6.0 MemorySegmentIndexInput.java:404, :962), so a backport needs the same change with those signatures.
• No benchmark was run.

What the checks validated

• Gradle in a scratch worktree of 645d4fe14a20, seed DEADBEEF: without the fix the repro fails with assertions on and off; with it the repro passes with assertions on and off (-Ptests.asserts=false), 0 skipped.
• With the fix, the closest suites pass: TestMMapDirectory (70), TestMultiMMap (73), TestLucene90CompoundFormat (36) and the repro: 180 tests, 0 failures, 1 skipped (testLargeCFS, @Nightly).
• :lucene:core:checkGoogleJavaFormat, ecjLintMain and ecjLintTest pass with the repro test in place.
• In the study, the same diff alone passed the verifiers' own tests (including an end-to-end compound-file test) and 178 upstream tests with -ea (the refute reviewer's own tests also with -da; the study's -da run of the 178 tests combined it with another fix of the same unit). A probe shows it advises [524288, 1048576) in the repro.

In slice(..., IOContext), call advise(offset, length, ...) on the input being sliced instead of slice.advise(0, slice.length, ...).

Trade-offs

One line, but correct only when the parent's offset is 0 (a whole file, as in the compound reader); a slice of a multi-chunk slice would still be advised wrongly, and it leaves the trap the #14109 review described. Not validated.

Advise only pages fully inside the inner file, as agreed as a follow-up in the #16883 review.

Trade-offs

The helper is shared with prefetch(), which needs outward rounding (jimczi, #16883), so it needs a parameter or a second helper. It does not fix this bug on its own. Not validated.

Method: semi-formal (a hypothesis formed while modelling, then witnessed by the solver). A hunt unit modelled the chunk arithmetic of MemorySegmentIndexInput (seek, slice, prefetch and advise ranges) in Z3 bit-vector arithmetic, 44 queries.

• While writing the model the AI tool noted that MultiSegmentImpl overrides prefetch() (the #14109 fix) but not advise(). One query, declared expect="sat" with that reasoning in its comment, was SAT: the range slice(..., IOContext) advises starts at the chunk start, before the slice. All 44 queries matched their declared expectation; it is the only one about advise(), and the prefetch range proofs hold (the other counterexample queries concern different findings of the same unit).
• Because the hypothesis came before the solver's answer, the finding is classed as semi-formal, not formal.
• The defect was then executed on the stock jars by reading the per-mapping random-read flag (rr) from /proc/self/smaps, and reproduced by every verifier with its own test.