A stored string of about 1 GiB of non-Latin-1 text is accepted by IndexWriter but cannot be read back

IndexWriter limits a stored string by its length in characters: it rejects only strings longer than IndexWriter.MAX_STORED_STRING_LENGTH, about 715.8 million chars, a bound derived from the worst-case UTF-8 encode buffer. Reading the field back goes through DataInput.readString, which calls the JDK's new String(bytes, 0, length, UTF_8). Since JDK 9 that constructor sizes its UTF-16 buffer by the UTF-8 byte count, and on JDK 21 and 25 it refuses 2^30 - 1 or more with OutOfMemoryError before it allocates the UTF-16 buffer (it has only copied the input bytes).

So a stored field of 357,913,941 CJK chars (1,073,741,823 bytes, about half of the accepted maximum) is indexed and committed, and then every StoredFields#document call on that document throws java.lang.OutOfMemoryError: UTF16 String size is 1073741823, should be less than 1073741823, with a 4, 6 or 10 GB heap alike; one char fewer reads back. A visitor-path merge (executed: addIndexes into an index with other field numbers) and, with an index sort, the flush of the segment that holds the document decode the field too, throw the same Error and close IndexWriter. The bytes on disk are valid: a read-side fix in DataInput.readString that decodes large values with Lucene's own UnicodeUtil.UTF8toUTF16 makes existing indexes of the default stored-fields format readable. It needs about 1 GiB in one field with some non-Latin-1 text, so it is rare; every release run on JDK 25, 5.5.5 to 10.6.0 and main, fails.

1. The write check counts chars. IndexingChain throws IllegalArgumentException("stored field ... is too large (... characters) to store") only when storedValue.getStringValue().length() > IndexWriter.MAX_STORED_STRING_LENGTH (IndexingChain.java:1491-1503). The constant is ArrayUtil.MAX_ARRAY_LENGTH / UnicodeUtil.MAX_UTF8_BYTES_PER_CHAR (IndexWriter.java:260-262), 715,827,877 on these JVMs: the largest string whose worst-case UTF-8 form fits in one byte[]. It is an encode-side bound.
2. The writer stores UTF-8. Lucene90CompressingStoredFieldsWriter.writeField(FieldInfo, String) calls bufferedDocs.writeString(value) (Lucene90CompressingStoredFieldsWriter.java:323-328). bufferedDocs is a ByteBuffersDataOutput (:95, :124), whose writeString override (ByteBuffersDataOutput.java:414-451) writes the format DataOutput.writeString documents (DataOutput.java:263-270): a vInt byte length (:437-438), then the UTF-8 bytes from UnicodeUtil.UTF16toUTF8 (for a value this long through writeLongString, :446, :567-581). A 1,073,741,823-byte value is written and committed without error.
3. Every read in the default format decodes with the JDK constructor. readField handles a STRING field with visitor.stringField(info, in.readString()) (Lucene90CompressingStoredFieldsReader.java:295-296), so the value is decoded before any visitor sees it; a visitor can only avoid that by answering NO or STOP in needsField (:706-718). DataInput.readString returns new String(bytes, 0, length, StandardCharsets.UTF_8) (DataInput.java:226-231). The non-default SimpleText codec decodes on its own with the same JDK constructor (SimpleTextStoredFieldsReader.java:165; not run).
4. The JDK refuses by byte count. For a result with non-Latin-1 chars, JDK 25's UTF-8 constructor calls StringUTF16.newBytesFor(length) with the UTF-8 byte count, and StringUTF16.newBytesLength throws OutOfMemoryError when that is >= Integer.MAX_VALUE >> 1 (2^30 - 1), before it allocates the UTF-16 buffer; by then it has only copied the input bytes (JDK 25 String.java:568 copies, :597 calls newBytesFor, StringUTF16.java:53-60 throws; the executed stack shows the frames at :597 and in StringUTF16). The decoded string, 357,913,941 chars, would itself fit. JDK 8 decoded into a char[len] and had no such cap.
5. Where it surfaces. StoredFields#document; visitor-path merges (MergeStrategy.VISITOR when field numbers differ, Lucene90CompressingStoredFieldsWriter.java:678-684, used at :637-640; executed for addIndexes(CodecReader), other merge configurations read from the code); the index-sort flush, which re-reads every stored field of the new segment through a CopyVisitor that loads every field (SortingStoredFieldsConsumer.java:109-139, :199-200); and CheckIndex's stored-field check. IndexWriter treats any Error as tragic and closes itself (class javadoc IndexWriter.java:162-165; doFlush :4428-4431; addIndexes(CodecReader...) :3391; prepareCommit :3879-3881, the path of the executed sorted commit()).

At least 2^30 - 1 UTF-8 bytes, at least one non-Latin-1 char, and (to stay under the char limit) more than 1.5 UTF-8 bytes per char on average: for example about 358M CJK chars, or about 537M Cyrillic chars (arithmetic, not run). Latin-1-only text above the boundary reads back: 536,870,912 x U+00E9 (1,073,741,824 bytes) passes, while the same text with one U+4E00 (1,073,741,825 bytes) fails.

• Who can be hit. Any application on the default codec and IndexWriterConfig that stores one text field (StoredField(String); a stored TextField holds the same value, not run, while a StringField this large is already rejected as an immense term) of at least 2^30 - 1 UTF-8 bytes containing a non-Latin-1 char and averaging more than 1.5 bytes per char. Large natural CJK text is enough.
• Scale. About 1 GiB in one field, and several GB of heap at index time (the test indexes it with a 4 GB heap). Near-GB stored fields do occur: LUCENE-7538 (#8589) started from an 800 MB field, and in the same thread (2017) a user reported files "which are 1GB huge". Common ingestion caps keep it out in practice (for example Elasticsearch's 100 MB default HTTP body limit), and nobody has reported it since JDK 9.
• What goes wrong. An Error on a document Lucene accepted, not a wrong answer. Unsorted index: the document is committed and stays unreadable; a search that loads all stored fields of its hits fails whenever it is among them; a visitor-path merge closes the writer. Index sort: the flush of the segment that holds it throws, commit() fails, the writer closes and every uncommitted document is lost (the last commit stays intact). CheckIndex does not see it at its default level; -level 2 -exorcise drops the whole segment, other documents included.
• Default vs opt-in. Default codec and configuration, no special API; the index-sort path is opt-in. 6.x to 8.x are hit only on a JDK 9+ runtime; 9.0.0 and later on every JDK they support (on JDK 11 and 17 from 2^30 bytes rather than 2^30 - 1; source reading, only JDK 25 was run).

The test sits in org.apache.lucene.index but uses only public API. It pins the default codec because LuceneTestCase randomizes it (the gradle "test environment" line still names the class-level random codec; IndexWriterConfig.setCodec overrides it for this writer). The input is the smallest that fails: the boundary run shows 1,073,741,820 bytes reading back and 1,073,741,823 failing, and the text must contain non-Latin-1 chars. Stock fails the same way with a 4 GB and a 10 GB heap; with the default 512 MB test heap it fails earlier, with "Java heap space", while building the string.

As a regression test it is the same test: it fails on stock and passes with either fix. Because it needs a multi-GB heap, upstream would probably mark it @Monster, like Test4GBStoredFields, or @Nightly. Under gradle's default test JVM arguments it passes with 6g with the recommended fix and needs 8g with the alternative one. All runs were on Linux aarch64 with OpenJDK 25.0.3.

TestStoredStringOver1GiBUtf8 > testReadBackStoredStringOf2Pow30Minus1Utf8Bytes FAILED
    java.lang.AssertionError: document(0) threw java.lang.OutOfMemoryError: UTF16 String size is 1073741823, should be less than 1073741823
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:3E746858805BB5]:0)
        at org.apache.lucene.index.TestStoredStringOver1GiBUtf8.testReadBackStoredStringOf2Pow30Minus1Utf8Bytes(TestStoredStringOver1GiBUtf8.java:43)

        Caused by:
        java.lang.OutOfMemoryError: UTF16 String size is 1073741823, should be less than 1073741823
            at java.base/java.lang.StringUTF16.newBytesLength(StringUTF16.java:58)
            at java.base/java.lang.StringUTF16.newBytesFor(StringUTF16.java:48)
            at java.base/java.lang.String.<init>(String.java:597)
            at java.base/java.lang.String.<init>(String.java:546)
            at org.apache.lucene.store.DataInput.readString(DataInput.java:230)
            at org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader.readField(Lucene90CompressingStoredFieldsReader.java:296)
            at org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsReader.document(Lucene90CompressingStoredFieldsReader.java:708)
            at org.apache.lucene.index.CodecReader$1.document(CodecReader.java:102)
            at org.apache.lucene.index.BaseCompositeReader$2.document(BaseCompositeReader.java:190)
            at org.apache.lucene.index.StoredFields.document(StoredFields.java:68)
            at org.apache.lucene.index.TestStoredStringOver1GiBUtf8.testReadBackStoredStringOf2Pow30Minus1Utf8Bytes(TestStoredStringOver1GiBUtf8.java:41)

• The contract. MAX_STORED_STRING_LENGTH: "Maximum length string for a stored field." (IndexWriter.java:260). StoredFields#document "Returns the stored fields of the n-th Document" and declares only CorruptIndexException and IOException (StoredFields.java:48-62). DataOutput.writeString and DataInput.readString are documented as a pair (@see, DataOutput.java:263-270, DataInput.java:221-225): the writer side accepts what the reader side refuses.
• The limit's purpose. LUCENE-7538 (#8589, 6.4.0): "Asking IndexWriter to store a too-massive text field now throws IllegalArgumentException instead of a cryptic exception that closes your IndexWriter" (lucene/CHANGES.txt:8734-8736). In the same issue Michael McCandless answered a user asking for the size limit: "see IndexWriter.MAX_STORED_STRING_LENGTH for the limit; you'll have to either not store that huge file, or break it up across several documents." The visitor merge and the sorted flush here end in the writer-closing outcome that change replaced with an up-front exception.
• The format's documented limit is larger. The default stored-fields format excludes only documents larger than 2^31 - 2^14 bytes (Lucene90StoredFieldsFormat.java:96-99); this document is about 2^30 bytes. No javadoc names a lower, content- or JDK-dependent limit.
• No test covers it. The only near-limit test, TestIndexWriter.testMassiveField, is @Ignore("requires running tests with biggish heap") and checks only the rejection above the limit (TestIndexWriter.java:3066-3095).
• History. The limit was chosen in 2016, when Lucene ran on Java 8, whose decoder had no 2^30 cap; JDK 9's compact strings halved the maximum size of a non-Latin-1 String (JDK-8190429, Won't Fix). The two limits stopped matching then; nothing in the history shows the gap was considered (the Java 8 reading is an inference, not executed).

The strongest case (the verification reviewers rated it weak; the later fact check by another model family rated it moderate, mainly because a 1 GiB field is operationally extreme): this is a JDK limit, not a Lucene bug; the constant bounds the encode buffer and never promised that every accepted string reads back; gigabyte stored fields are unreasonable (in #8589, Erick Erickson wrote "I question whether it's reasonable to even try to index documents this large"); and IndexWriter's javadoc says it closes itself on any Error.

Why it fails: Lucene chooses both the bound it accepts and the decode call. The cap is a fixed property of every JDK that main supports and, once the heap holds the bytes, does not depend on the heap size, so the failure is deterministic, not an occasional resource problem. The decoded 357.9M-char string is representable (the writer held it) and UnicodeUtil.UTF8toUTF16 decodes it (see the fix). The committer who added the constant presented it to users as the limit of what can be stored; "unreasonable" is an opinion in a comment, not documentation, and the user in that thread answered that their 1 GB files are a valid use case. "Closes on any Error" explains the second symptom, not the first Error. The extreme trigger lowers the priority, which its Low rating reflects; it does not make the behaviour correct.

Two read-side fixes were run through gradle in a scratch worktree of 645d4fe14a20 (OpenJDK 25.0.3, seed DEADBEEF, gradle's default test JVM arguments). Both change only DataInput.readString and only for byte lengths of 2^30 - 1 or more. Every stored-string read of the default stored-fields format goes through readString, so they fix all of its executed paths at once (StoredFields#document, the visitor merge, the index-sort flush, CheckIndex) with no format change. They do not touch the non-default SimpleText codec, which decodes on its own (SimpleTextStoredFieldsReader.java:165), and were tested on the default codec only. The closest suites never reach the new branch (nothing in them is near 1 GiB); the repro and the shadow runs do.

In DataInput.readString (DataInput.java:226-231), plus import org.apache.lucene.util.UnicodeUtil;. One counting pass over the bytes finds the UTF-16 length (one unit per lead byte, two for a 4-byte sequence), then Lucene's decoder fills a char[] of exactly that size.

     readBytes(bytes, 0, length);
-    return new String(bytes, 0, length, StandardCharsets.UTF_8);
+    if (length < (Integer.MAX_VALUE >> 1)) {
+      return new String(bytes, 0, length, StandardCharsets.UTF_8);
+    }
+    // String(byte[], UTF_8) sizes its UTF-16 buffer by the byte count and refuses 2^30 - 1 or
+    // more units, although the decoded string is shorter: count the UTF-16 units, then decode
+    int numChars = 0;
+    for (int i = 0; i < length; i++) {
+      final int b = bytes[i] & 0xff;
+      if ((b & 0xc0) != 0x80) { // a lead byte; 4-byte sequences decode to a surrogate pair
+        numChars += b >= 0xf0 ? 2 : 1;
+      }
+    }
+    final char[] chars = new char[numChars];
+    return new String(chars, 0, UnicodeUtil.UTF8toUTF16(bytes, 0, length, chars));
   }

Trade-offs

• One extra pass over the bytes, only on this rare path. The temporary char[] is 716 MB for this value, next to the 1 GiB byte[] that readString already allocates. Latin-1-only values of 2^30 - 1 or more bytes, which read back today, take the new branch too, so their temporary memory changes as well (a char[] at 2 bytes per char instead of the JDK's own copies); that case was not run with this fix and its heap use was not measured.
• UnicodeUtil.UTF8toUTF16 does not validate UTF-8 and can throw on invalid input (its javadoc), unlike the JDK decoder, which substitutes malformed input. Lucene's own writers produce valid UTF-8 (the stored-fields writer's ByteBuffersDataOutput.writeString encodes with UnicodeUtil.UTF16toUTF8, which writes unpaired surrogates as U+FFFD) and index files are trusted. But readString is a public DataInput method, so malformed input of that size from any other source (a corrupt file, or data another writer produced) would behave differently.

What the checks validated

• Stock: the repro fails with assertions on and off (-Ptests.heapsize=6g).
• Fix: the repro passes at 6g with assertions on and off.
• The closest suites (all of org.apache.lucene.store.*, TestLucene90StoredFieldsFormat, its HighCompression and MergeInstance variants, TestCompressingStoredFieldsFormat, TestStoredFieldsInt, TestSortingStoredFieldsConsumer, TestStoredFieldsConsumer, TestStoredFieldsMergeReadAdvice, TestIndexWriter, TestUnicodeUtil): 38 classes, 996 tests, 0 failures, 26 skipped.
• :lucene:core:check -x test passes; with the repro test in the tree, checkGoogleJavaFormat, ecjLintTest and forbiddenApisTest pass. The full core suite was not run.
• Shadowed onto the stock jars: the hunt's 3 tests, including the addIndexes visitor merge, pass; in the consequences probe all 9 hits load fully and, with an index sort, commit() returns normally and the writer stays open.

The same branch in DataInput.readString, without the counting loop.

+    if (length < (Integer.MAX_VALUE >> 1)) {
+      return new String(bytes, 0, length, StandardCharsets.UTF_8);
+    }
+    final char[] chars = new char[length];
+    final int numChars = UnicodeUtil.UTF8toUTF16(bytes, 0, length, chars);
+    return new String(chars, 0, numChars);

Trade-offs

Simplest, but the char[length] temporary costs 2 bytes per UTF-8 byte, 2 GiB for this value. Under gradle's default test JVM arguments (-XX:+UseParallelGC) the repro then runs out of heap at 6g and needs 8g.

What the checks validated

Gradle: the repro passes at -Ptests.heapsize=8g with assertions on and off; the same closest suites pass (38 classes, 996 tests, 0 failures, 26 skipped); :lucene:core:check -x test passes. Shadowed onto the stock jars it passed the hunt's 3 tests, the refute reviewer's own repro, and the minimal test at 6 GB with the standalone runner.

Check the UTF-8 byte length in IndexingChain (only for strings longer than 357,913,940 chars, so normal strings pay nothing), or lower MAX_STORED_STRING_LENGTH to 357,913,940 chars.

Trade-offs

Gives the up-front IllegalArgumentException LUCENE-7538 intended, also for the sorted flush, but leaves already written documents unreadable. A plain byte-length rule, or the lower constant, newly rejects Latin-1 or ASCII strings that read back today; an exact rule would mirror a JDK implementation detail. Complementary to a read-side fix, which alone removes every executed failure.

Method: non-formal (found by execution), a hunt unit on the compressing stored-fields format.

• A separate finding of the same unit, an int overflow in the stored-fields writer's slice loop for chunks over about 2^31 bytes (LFV-186), was being validated with a patched writer that stored a 2.1 GB String. The read-back failed in DataInput.readString with the JDK's "UTF16 String size" error.
• An AI tool then measured the boundary on the stock jars and wrote a deterministic JUnit with a control on each side: 357,913,940 chars (1,073,741,820 bytes) read back, 357,913,941 chars (1,073,741,823 bytes) fail.
• The unit's Z3 model of the stored-fields chunk arithmetic (an obligation on MAX_STORED_STRING_LENGTH) gives context only; it did not produce this signal.