The test framework's RAMOnlyPostingsFormat cannot flush a DOCS-only field: NegativeArraySizeException: -1

RAMOnlyPostingsFormat, the in-memory postings format of lucene-test-framework (SPI name RAMOnly), cannot write a field indexed with IndexOptions.DOCS, the index option of StringField and KeywordField. Its writer passes the "no frequencies" marker freq = -1 to its own startDoc, the same convention as Lucene's push-style postings writers, but its per-document record RAMDoc sizes its positions array with new int[freq]. So the first flush that contains such a field throws NegativeArraySizeException: -1, an aborting exception that closes the IndexWriter. One document with one StringField is enough, with assertions on or off. The array also serves as the format's frequency (freq() returns its length); the -1 marker arrived in 2012 (LUCENE-4248), after the array was written, and RAMDoc was never adapted to it (LUCENE-5268 later rewrote the writer and kept the -1). Only tests that pick this format are hit: Lucene's randomized codec never picks it, and no Lucene test actually writes a field with it.

// :227-229, :270-278  RAMFieldsConsumer.write
boolean writeFreqs = indexOptions.subsumes(IndexOptions.DOCS_AND_FREQS);   // false for DOCS
...
int freq;
if (writeFreqs) {
  freq = postingsEnum.freq();
  totalTermFreq += freq;
} else {
  freq = -1;                                  // "no frequencies" marker
}
postingsWriter.startDoc(docID, freq);

// :350-351  RAMPostingsWriterImpl.startDoc
current = new RAMDoc(docID, freq);

// :168-170  RAMDoc
positions = new int[freq];                    // new int[-1] -> NegativeArraySizeException: -1

// :481-484  RAMDocsEnum.freq: the array doubles as the frequency
return current.positions.length;

• Where the -1 comes from: write()'s per-document loop (RAMOnlyPostingsFormat.java:262-297) matches, statement for statement, the one in the core PushPostingsWriterBase (PushPostingsWriterBase.java:130-164), including freq = -1 when frequencies are omitted (PushPostingsWriterBase.java:142); that class's startDoc contract reads "freq will be -1 when term frequencies are omitted for the field" (:177-181). RAMOnly's RAMPostingsWriterImpl does not extend that class but follows the same convention; its RAMDoc does not.
• Why the writer closes: the exception is thrown inside the DWPT flush, which calls onAbortingException(t) (DocumentsWriterPerThread.java:585-587), so the IndexWriter hits a tragic event. This doc's probe sees isOpen() == false after the failed flush; the study verifier also checked that getTragicException() is the same exception.
• Same root, sibling symptom (context, not part of the claim): a DOCS_AND_FREQS field allocates new int[freq] but never adds positions, so with -ea finishDoc's assert posUpto == current.positions.length (:370-372) fails the flush; with -da it works. The recommended fix covers both.

• Routes: -Dtests.postingsformat=RAMOnly makes the test framework use PostingsFormat.forName("RAMOnly") for every field (SetupAndRestoreStaticEnv.java:147-158), or a test sets it itself (for example with TestUtil.alwaysPostingsFormat). The package is exported and the format registered as a PostingsFormat provider (module-info.java:45, :75; META-INF/services/org.apache.lucene.codecs.PostingsFormat:17), so downstream tests can use it.
• Not default: RandomCodec never picks it (RandomCodec.java:236-259). Its only user in Lucene's sources, TestExternalCodecs, routes every field except field1, field2 and id to it, but its documents contain only those three fields (TestExternalCodecs.java:44-51, :72-83). No BasePostingsFormatTestCase subclass is for RAMOnly.
• Scale: one document with one StringField; deterministic; the first flush fails.
• Effect: a NegativeArraySizeException and a closed writer in a test. No index damage, no wrong results, no production code path.

• Smallest input: one document, one DOCS-only term. The failure is inside the postings write, before a compound file is built or a reader is opened; setUseCompoundFile(false) and a plain ByteBuffersDirectory keep the fixed test from tripping the class's separate defects (compound-file header check, CheckIndex on close).
• Assertions: pass -Ptests.asserts explicitly. Lucene's build enables assertions for about 75% of seeds, and with seed DEADBEEF its default is off (TestsAndRandomizationPlugin.java:240-249); a run without the flag printed -Ptests.asserts=false in its reproduce line.
• Releases: the same test fails on the 10.5.2 release jars with assertions on and off; it compiles from 9.1.0 on, and fails on all 34 targets 9.1.0 to 10.6.0 and main. A portable probe with the same input fails on every release from 4.0.0-BETA.
• As the regression test: this same method, in a new lucene/test-framework/src/test/org/apache/lucene/tests/codecs/ramonly/ (next to the existing codecs/asserting/ tests). With the fix, :lucene:test-framework:check -x test passes with the test as is.

mkdir -p $LUCENE/lucene/test-framework/src/test/org/apache/lucene/tests/codecs/ramonly
cp $REPRO/TestRAMOnlyPostingsFormat.java $LUCENE/lucene/test-framework/src/test/org/apache/lucene/tests/codecs/ramonly/
cd $LUCENE
./gradlew :lucene:test-framework:test --tests org.apache.lucene.tests.codecs.ramonly.TestRAMOnlyPostingsFormat -Ptests.seed=DEADBEEF -Ptests.asserts=true
# assertions off: -Ptests.asserts=false (same failure)

# gradle, main 645d4fe14a20, seed DEADBEEF, -Ptests.asserts=true
TestRAMOnlyPostingsFormat > testDocsOnlyField FAILED
    java.lang.AssertionError: expected flush of a DOCS-only field to succeed, got java.lang.NegativeArraySizeException: -1
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:6082079233177FCB]:0)
        at org.apache.lucene.tests.codecs.ramonly.TestRAMOnlyPostingsFormat.testDocsOnlyField(TestRAMOnlyPostingsFormat.java:46)

        Caused by:
        java.lang.NegativeArraySizeException: -1
            at org.apache.lucene.tests.codecs.ramonly.RAMOnlyPostingsFormat$RAMDoc.<init>(RAMOnlyPostingsFormat.java:170)
            at org.apache.lucene.tests.codecs.ramonly.RAMOnlyPostingsFormat$RAMPostingsWriterImpl.startDoc(RAMOnlyPostingsFormat.java:351)
            at org.apache.lucene.tests.codecs.ramonly.RAMOnlyPostingsFormat$RAMFieldsConsumer.write(RAMOnlyPostingsFormat.java:278)
            at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:162)
            at org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:133)
            ...
            at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4357)
            at org.apache.lucene.tests.codecs.ramonly.TestRAMOnlyPostingsFormat.testDocsOnlyField(TestRAMOnlyPostingsFormat.java:43)
1 test completed, 1 failed
BUILD FAILED
# -Ptests.asserts=false: the same failure and stack
# 10.5.2 release jars, assertions on and off: the same message, RAMDoc.<init>(RAMOnlyPostingsFormat.java:169)
# with the recommended fix: BUILD SUCCESSFUL, 1 test, 0 failures (assertions on and off)

• No documented limitation: the class javadoc says "Stores all postings data in RAM, but writes a small token (header + single int) ... NOTE: this codec sorts terms by reverse-unicode-order!" (RAMOnlyPostingsFormat.java:56-61); the package javadoc says "Codec for testing that never writes to disk." (package-info.java:18).
• The designed limitation is explicit: throw new UnsupportedOperationException("this codec cannot index offsets") (:211-215). DOCS gets no such check; it fails on an array allocation.
• The writer handles DOCS on purpose: it computes writeFreqs and branches on it (:228, :234-235, :271-276); the crash is in the storage class it hands the marker to.
• The read contract: PostingsEnum#freq "Returns term frequency in the current document, or 1 if the field was indexed with IndexOptions#DOCS" (PostingsEnum.java:74-82). CheckIndex enforces it ("When a field didn't index freq, it must consistently 'lie' and pretend that freq was 1", CheckIndex.java:1765-1777) and requires totalTermFreq == docFreq for such fields (:1695-1705, :1935-1937). FieldsConsumer#write puts no restriction on index options (FieldsConsumer.java:47-64).
• History: LUCENE-4248 (2012) made the indexing chain pass -1 instead of 0 for omitted frequencies and documented it; it did not touch RAMOnly. No commit message, comment, CHANGES or MIGRATE entry mentions a RAMOnly index-options limitation.

The strongest case for "intended": RAMOnly is a test-only codec (behind the @lucene.experimental PostingsFormat API) that nothing in Lucene's own suite exercises; the only in-tree route is an unusual system property; and the study verifier found other, separate breakages in the class (with compound files on, the default, any field fails because its 17-byte .id file has no index header or footer; positions without payloads are stored as -1; RAMDocsEnum.docID() throws before nextDoc()). So one could call it dead code that simply does not support DOCS. The cross-model refute review (OpenAI GPT-5.6) weighed exactly this status and still judged the bug real with high confidence.

Why it fails: the class is shipped, exported and SPI-registered in lucene-test-framework, so downstream tests can use it; nothing documents a DOCS restriction while the one real restriction, offsets, is enforced explicitly; and the crash comes from the format's own writer feeding its own storage the documented push-API marker. Low use lowers the severity, not the correctness. Fixing or removing the class is upstream's choice; either resolves the bug.

One fix was built and validated with gradle; the alternatives were not run.

+10/-7 lines in RAMOnlyPostingsFormat.java, no core change. The full diff (it applies to 645d4fe14a20):

--- a/lucene/test-framework/src/java/org/apache/lucene/tests/codecs/ramonly/RAMOnlyPostingsFormat.java
+++ b/lucene/test-framework/src/java/org/apache/lucene/tests/codecs/ramonly/RAMOnlyPostingsFormat.java
@@ -162,12 +162,14 @@
 
   static class RAMDoc implements Accountable {
     final int docID;
+    final int freq;
     final int[] positions;
     byte[][] payloads;
 
-    public RAMDoc(int docID, int freq) {
+    public RAMDoc(int docID, int freq, boolean hasPositions) {
       this.docID = docID;
-      positions = new int[freq];
+      this.freq = freq;
+      positions = new int[hasPositions ? freq : 0];
     }
 
     @Override
@@ -272,10 +274,11 @@
               freq = postingsEnum.freq();
               totalTermFreq += freq;
             } else {
-              freq = -1;
+              freq = 1;
+              totalTermFreq += freq;
             }
 
-            postingsWriter.startDoc(docID, freq);
+            postingsWriter.startDoc(docID, freq, writePositions);
             if (writePositions) {
               for (int i = 0; i < freq; i++) {
                 int pos = postingsEnum.nextPosition();
@@ -347,8 +350,8 @@
       this.term = term;
     }
 
-    public void startDoc(int docID, int freq) {
-      current = new RAMDoc(docID, freq);
+    public void startDoc(int docID, int freq, boolean hasPositions) {
+      current = new RAMDoc(docID, freq, hasPositions);
       term.docs.add(current);
       posUpto = 0;
     }
@@ -480,7 +483,7 @@
 
     @Override
     public int freq() throws IOException {
-      return current.positions.length;
+      return current.freq;
     }
 
     @Override

// RAMDoc
final int freq;
public RAMDoc(int docID, int freq, boolean hasPositions) {
  this.docID = docID;
  this.freq = freq;
  positions = new int[hasPositions ? freq : 0];
}
// RAMFieldsConsumer.write, field without frequencies
freq = 1;
totalTermFreq += freq;
postingsWriter.startDoc(docID, freq, writePositions);
// RAMDocsEnum.freq()
return current.freq;

Trade-offs

• Separates the array's two jobs (frequency and positions storage), returns 1 for DOCS as PostingsEnum#freq and CheckIndex require (CheckIndex.java:1765-1777), and keeps totalTermFreq == docFreq for DOCS terms (:1695-1705, :1935-1937; TermsEnum#totalTermFreq is "the sum of the freq() for each doc").
• Also fixes the sibling DOCS_AND_FREQS assertion in finishDoc.
• RAMDoc gains one int per posting (test-only memory).
• The class keeps its other, separate defects (compound files, positions without payloads, docID() before nextDoc()); this fix does not address them.

What the checks validated

• Gradle, scratch worktree of 645d4fe14a20, OpenJDK 25.0.3, seed DEADBEEF: the minimal test fails before the change and passes after it, with assertions on and off.
• :lucene:test-framework:test (all of it, with the new test): 643 tests (Gradle's count), 33 skipped, 0 failures; :lucene:core:test --tests org.apache.lucene.TestExternalCodecs: 1 test, 0 failures (assertions on).
• :lucene:test-framework:check -x test (checkGoogleJavaFormat, ecjLint, forbiddenApis, checkLicenses): passes.
• A probe with the fix reads back docFreq=1 totalTermFreq=1 freq=1 for a StringField, and freqs 2 and 1 for a DOCS_AND_FREQS field "a a b" (this doc's run with assertions on; the independent fact check re-ran it with them on and off: same values); without it both flushes fail with -ea, and the DOCS one also with -da. The one-method test itself checks only that the flush no longer throws; the read-back values come from this probe.
• Earlier and independently: a standalone fix check (pass with -ea and -da), and the cross-model reviewer's own two tests and TestExternalCodecs with the same file.

positions = new int[Math.max(freq, 0)]; in RAMDoc stops the exception with one line. Not run.

Trade-offs

freq() then returns 0 for DOCS documents (CheckIndex requires 1), DOCS terms keep totalTermFreq = 0 (CheckIndex requires docFreq), and DOCS_AND_FREQS still trips the assertion. Not recommended.

Throw UnsupportedOperationException for fields without frequencies, as for offsets; or delete RAMOnlyPostingsFormat with its SPI and module entries. Not run.

Trade-offs

Rejecting gives a clear message but makes the most basic index option unsupported. Removal fits a class nothing in Lucene's suite exercises, but it is an API removal in the test framework and a maintainers' call.

• First signal: the study's fuzzing and property-based run (2026-10-22, main 7cecf79523) compared Lucene against independent oracles, including a second implementation inside Lucene: 15 postings formats. A probe that indexes the same documents and runs the same queries through every registered format found that all agreed except RAMOnly, whose flush threw NegativeArraySizeException on a StringField.
• Verification: the study verifier wrote its own 9-test class, a plain main() run with -ea and -da, a run through -Dtests.postingsformat=RAMOnly, and an experimental patched copy that confirmed the root cause; it reproduced the crash on main and on the 10.5.2 jars.
• Minimal repro (2026-10-29): one test method, one StringField, public API only; fails on main 645d4fe14a20 with assertions on and off.