LazyDocument's value-count assertion is reversed: with assertions on, reading a multi-valued lazy field before its last getField call throws AssertionError

LazyDocument.getField(FieldInfo) (module lucene/misc) returns a new placeholder on every call; its NOTE asks callers to call it once per stored value of the field, in stored order. The first read of a placeholder calls fetchRealValues, which loads the document and fills every placeholder of that field created so far. Before filling them it asserts realValues.length <= lazyValues.size() with the message "More lazy values then real values for field: " (LazyDocument.java:113-114), but the loop right after it reads realValues[i] for every placeholder (:116-121); since getField only ever adds non-null placeholders (:77-78), the loop is safe exactly when lazyValues.size() <= realValues.length: the condition is the reverse of what the message and the loop need. With assertions on, reading the first value of a two-valued field before the second getField call throws AssertionError, although getField resets the cached document whenever more placeholders are requested after a read (:80-86) and with assertions off the same calls return the right values. The case the message names, more getField calls than stored values, passes the assertion and fails in the loop with ArrayIndexOutOfBoundsException. It came with SOLR-4589 (2013, first released in 4.2.1) and is unchanged up to 10.6.0 and main. Production runs (assertions off) are never wrong; the fix is to swap the two sides.

• Placeholder (lazy field): the LazyDocument.LazyField that getField returns. lazyValues is the list of placeholders created so far for one field; realValues is every stored value of that field in the loaded document.
• Over-request: calling getField more often than the field has stored values. The getField NOTE forbids it ("must be called once for each value of the field name specified in sequence that the values exist", LazyDocument.java:61-63).

1. getField (:68-88) appends a new placeholder to the field's list (:71-78) and sets doc = null (:85), with the comment "edge case: if someone asks this LazyDoc for more LazyFields after other LazyFields from the same LazyDoc have been actuallized, we need to force the doc to be re-fetched so the new LazyFields are also populated" (:81-84).
2. A read such as stringValue() (:170-172) calls getRealValue() (:148-157), which calls fetchRealValues while the value is unset (:149-150). fetchRealValues (:107-122) loads the document if needed (getDocument(), :95-104), takes lazyValues (:110) and realValues = d.getFields(name) (:111).
3. It asserts realValues.length <= lazyValues.size() with the message "More lazy values then real values for field: " (:113-114), then sets realValue = realValues[i] for every i < lazyValues.size() (:116-121). The loop needs lazyValues.size() <= realValues.length (it skips null placeholders, but getField never adds one, :77-78); the assertion checks the reverse.
4. So with assertions on it rejects a read made while fewer placeholders than stored values exist, which the loop handles fine, and it lets through the over-request, which the loop does not handle: ArrayIndexOutOfBoundsException at :119, with assertions on as well as off.

Stored f = v0, v1. After the first getField(f): 1 placeholder, 2 stored values; the first read asserts 2 <= 1 and fails. With assertions off the loop fills placeholder 0 with v0. The second getField(f) sets doc = null; its read reloads the document, asserts 2 <= 2 and fills both placeholders: v1. Only the first read ever fails, because after the field's last getField call the two counts are equal. Over-request: 1 stored value, 2 calls: 1 <= 2 holds and the loop throws Index 1 out of bounds for length 1.

The same assertion fires when the read happens inside a StoredFieldVisitor visit, before the visitor has requested the field's later values (an independent probe, on main and 10.5.2).

• Who is hit. Code that uses LazyDocument (no other Lucene class does: in the tracked sources only LazyDocument.java and TestLazyDocument.java mention it) and runs with assertions enabled and reads a lazy value of a multi-valued field before the field's last getField call, directly or from inside a StoredFieldVisitor. Assertions are typically enabled in test runs; Lucene's own build enables them for about 75% of test seeds.
• What happens. A deterministic AssertionError at that read. With assertions off the values are right: the hunt's bounded-exhaustive harness found 0 wrong values and 0 exceptions in 2,088,699 checks (1,986,359 value reads plus one whole-document check per schedule).
• The guard does not guard. The misuse its message names (more getField calls than stored values) reaches an ArrayIndexOutOfBoundsException instead of the intended assertion message, with assertions on.
• Default vs opt-in. An opt-in module; assertions are off by default in the JVM. No scale is needed: two stored values.
• How likely. Rare. The usual flow, and Solr's (its stored-field visitor requests every value during the visit and reads afterwards, by reading its source), requests every value first. No public report in the 13 years since 4.2.1.

The test sits in org.apache.lucene.misc.document, LazyDocument's package, by Lucene test convention; it uses only public API (LazyDocument(IndexReader, int), getField(FieldInfo) and stringValue() on the IndexableField it returns; it never names LazyDocument.LazyField or getDocument(), which are marked @lucene.internal). The outcome does not depend on the seed or on the test framework's random codec.

Not shown by this one test, all from the same assertion and executed elsewhere: a read in the middle of a StoredFieldVisitor visit, and the over-request that reaches ArrayIndexOutOfBoundsException.

As the regression test: the same test unchanged, for example as a new method of TestLazyDocument. It fails on stock main with assertions on and passes with fix A with assertions on and off (gradle). :lucene:misc:check -x test passes with fix A and this test in place.

cp TestLazyDocumentReadBeforeLastValue.java $LUCENE/lucene/misc/src/test/org/apache/lucene/misc/document/
cd $LUCENE && ./gradlew :lucene:misc:test --tests org.apache.lucene.misc.document.TestLazyDocumentReadBeforeLastValue -Ptests.seed=DEADBEEF -Ptests.asserts=true
# assertions off: -Ptests.asserts=false instead (the test then passes: v0, v1)

# gradle, main 645d4fe14a20, -Ptests.seed=DEADBEEF -Ptests.asserts=true (with =false the test passes; the same on 10.6.0)
TestLazyDocumentReadBeforeLastValue > testReadFirstValueBeforeRequestingSecond FAILED
    java.lang.AssertionError: More lazy values then real values for field: f
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:73F8C0A28644819C]:0)
        at org.apache.lucene.misc.document.LazyDocument.fetchRealValues(LazyDocument.java:113)
        at org.apache.lucene.misc.document.LazyDocument$LazyField.getRealValue(LazyDocument.java:150)
        at org.apache.lucene.misc.document.LazyDocument$LazyField.stringValue(LazyDocument.java:171)
        at org.apache.lucene.misc.document.TestLazyDocumentReadBeforeLastValue.testReadFirstValueBeforeRequestingSecond(TestLazyDocumentReadBeforeLastValue.java:45)

1 test completed, 1 failed

• Message vs condition. "More lazy values then real values" means lazyValues.size() > realValues.length, the only state in which the loop at :116-121 can index past realValues. The condition rejects realValues.length > lazyValues.size() instead, which the loop handles fine.
• The code supports the order. getField resets the cached document for exactly this "edge case" (:81-85), and the next read fills the new placeholder; with assertions off the values are correct.
• Same commit, same style. The two assertions added beside it word their messages as the failure: assert hasBeenLoaded() : "field value was not lazy loaded" and assert realValue.name().equals(name()) : "realvalue name != name: ..." (:152-154).
• The author's stated intent (SOLR-4589). On 2013-03-16, before the patch, they wrote that a client app generating LazyField instances for the same field twice "will most likeley get an ArrayIndexOutOfBoundsError". On 2013-03-19, asked whether the re-fetch edge case happens in Solr: "no, but the way the public API for LazyDocument is defined, it is possible ... so i wanted to protect against it -- i included test randomization to verify that the edge case works." Nobody in the thread discusses the condition itself; reading it as reversed is an interpretation of the code. The quotes show that the author meant later getField calls after a read to work; they do not say whether such a call was expected for the same field or for another one.
• Never tested. Upstream TestLazyDocument requests every value inside the visitor before reading (TestLazyDocument.java:101-104, :203-215), and its step commented "use the same LazyDoc to ask for one more lazy field" builds a new LazyDocument for a different field (:161-165).
• History. 4.0.0 and 4.2.0 pass the same read order with assertions on; 4.2.1, whose only LazyDocument change is the SOLR-4589 backport, fails; 4.2.1 with only the condition flipped passes.

The strongest case for "intended" (the cross-model reviews and the adjudication rate it moderate; the independent reproducer, the devil's advocate and the audit rebut it without rating it): the getField NOTE ("must be called once for each value of the field name specified in sequence that the values exist", :61-63) can be read as "request every value before reading any"; the author called the API "really only usable from a StoredFieldVisitor"; Solr and the upstream test always request everything first. On that reading the condition is deliberate and only the message is wrong.

Why it fails: no javadoc, CHANGES.txt or MIGRATE.md text states a timing rule (none explicitly allows early reads either), and getField explicitly handles requests after a read, which the author said they wanted to protect. Even on the strict reading the assertion is wrong, because it lets through the over-request its message describes. Verdict: not intended; an inverted assertion of low impact. Residual risk: a maintainer may read the API as meant only for visitors; the re-fetch branch, the message and the AIOOBE argue against that reading.

Fix A was run through gradle in a scratch worktree of 645d4fe14a20; B and C are alternatives, not tested.

One line in fetchRealValues (LazyDocument.java:113-114), plus the message typo.

-    assert realValues.length <= lazyValues.size()
-        : "More lazy values then real values for field: " + name;
+    assert lazyValues.size() <= realValues.length
+        : "More lazy values than real values for field: " + name;

Trade-offs

• For every state getField can create (non-null placeholders only, LazyDocument.java:77-78) it accepts exactly the states the loop can handle and rejects the one it cannot, so the over-request now gets the assertion message instead of an ArrayIndexOutOfBoundsException when assertions are on (an independent reproducer's fix check).
• No change at all with assertions off. Under the strict reading of the getField NOTE, a read before the last getField call is no longer flagged; it returns the right values, so nothing is lost.

What the checks validated

• Gradle in a scratch worktree of 645d4fe14a20 (2026-11-01): the repro fails on stock main with -Ptests.asserts=true and passes with =false; with A it passes with both.
• With A: upstream TestLazyDocument 5 of 5 iterations with assertions on; the whole lucene/misc test suite with assertions on, 24 classes including the repro, 317 tests, 0 failures, 9 skipped (by the test sources 3 @Nightly and 6 that need an FSDirectory; the gradle log records only the count); :lucene:misc:check -x test (google-java-format, ecj lint of main and test, forbidden APIs) passes with A and the test in place.
• Earlier, the same fix: the finder's repro and upstream TestLazyDocument; an independent 3-test class 3/0 and upstream TestLazyDocument 5/5; the cross-model reviewers' own tests; the bounded-exhaustive harness with assertions on, 0 wrong values and 0 exceptions in 2,088,699 checks; 4.2.1 with the same flip.

Replace the assertion with if (lazyValues.size() > realValues.length) throw new IllegalStateException(...), so a caller that breaks the NOTE gets a clear message in production too.

Trade-offs

Changes behaviour with assertions off (an IllegalStateException instead of the ArrayIndexOutOfBoundsException), so it is a slightly bigger decision than a bug fix. Not tested.

assert lazyValues.size() == realValues.length.

Trade-offs

Fixes the over-request message but keeps rejecting the read-before-last-getField order that getField's re-fetch branch supports and that works with assertions off. Not recommended; not tested.

Method: semi-formal (bounded-exhaustive check of the stock jar). A hunt unit auditing the lucene/misc collectors, selectors and document helpers wrote a harness that builds every small document shape (1-3 stored fields of 1-3 values each and every interleaving of the values: 5,117 shapes; each field's value type is drawn at random from six stored types), calls getField once per stored value in stored order (the documented contract), and after every getField call reads a random subset of the placeholders created so far (20 schedules per shape), comparing each value with storedFields().document().

With assertions off: 2,088,699 checks (1,986,359 value reads plus one whole-document check per schedule, 102,340), 0 wrong values, 0 exceptions. With assertions on: 99,858 of the 102,340 schedules threw AssertionError: More lazy values then real values; the shortest was get(f#0) get(f#1) read(f@0) on a three-valued field. With the condition flipped, assertions on: 2,088,699 checks, 0 wrong values, 0 exceptions.

The finder reduced it to a public-API JUnit test. It was then reviewed by a reviewer from another model family (OpenAI GPT-5.6), reproduced independently, argued against by an intended-behaviour devil's advocate, judged, re-run and audited, and cut to the one-test repro on 645d4fe14a20.