String sort fields accept any missing value, silently treat it as STRING_FIRST, and an index sort with one cannot be reopened

For string sorts Lucene supports two missing values, SortField.STRING_FIRST and SortField.STRING_LAST. #15483 (10.4.0) replaced the validating setMissingValue with constructor parameters and kept the check only for Type.STRING. Since then new SortField(f, Type.STRING_VAL, reverse, x), new SortedSetSortField(f, reverse, selector, x) and KeywordField.newSortField(f, reverse, selector, x) store any x. The comparators and the index sorter of these fields only ask missingValue == STRING_LAST, so x acts as STRING_FIRST, while getMissingValue() and toString() report x.

As an index sort, SortedSetSortField.serialize writes "no missing value" for anything but the two sentinels. SortField.equals compares the missing value, so the next IndexWriter, opened with a fresh config holding the same sort, refuses the index with IllegalArgumentException: cannot change previous indexSort. In the probe no documents were lost, and removing the value from the sort opens the index again. Releases up to 10.3.2 are not affected, and on 10.4 to 10.6 the deprecated setter still rejects the value; on main (11.0) the constructors are the only path.

• Missing value: the fourth argument of new SortField(field, type, reverse, missingValue), new SortedSetSortField(field, reverse, selector, missingValue) and KeywordField.newSortField(field, reverse, selector, missingValue). It decides where documents without a value in the field sort. Numeric sorts take a substitute number; string sorts take one of two markers, SortField.STRING_FIRST or SortField.STRING_LAST (SortField.java:309-324). null means the default, which sorts like STRING_FIRST.
• STRING_VAL: a SortField type that compares the BytesRef values of a binary doc-values field. SortedSetSortField sorts a sorted-set field, which is what KeywordField.newSortField returns.
• Index sort: IndexWriterConfig.setIndexSort(sort) makes every segment store its documents in that order. Each segment records the sort it was written with, and a new IndexWriter checks that its config's sort matches every existing segment's.

1. SortField.validateField rejects a non-marker value only if (type == Type.STRING) (SortField.java:343-348). The SortedSetSortField constructor passes Type.CUSTOM to super and checks nothing itself (SortedSetSortField.java:74-81); KeywordField.newSortField passes the value on (KeywordField.java:188-193).
2. The comparators read only missingValue == STRING_LAST (SortField.java:521-523, SortedSetSortField.java:175-176), and so does the index sorter, both when a new segment is sorted at flush (IndexSorter.java:512-519) and at merge (:488-493): any other value sorts like STRING_FIRST.
3. SortedSetSortField.serialize writes 1 for STRING_FIRST, 2 for STRING_LAST and 0 otherwise (SortedSetSortField.java:124-135), so the stored sort has no missing value.
4. SortField.equals compares the missing value (SortField.java:451-461), so IndexWriter.validateIndexSort finds the config's sort unequal to the stored one and throws (IndexWriter.java:1213-1239). Each writer needs its own IndexWriterConfig with the same sort; reusing the first writer's config fails earlier, with IllegalStateException: do not share IndexWriterConfig instances across IndexWriters (IndexWriter.java:935-947). Within the first writer session nothing fails: two commits and a forceMerge(1) succeed.

The same comparison runs in addIndexes(Directory...) and addIndexes(CodecReader...) (IndexWriter.java:3152-3158, :3228-3250): adding an index written with the "zzz" sort to a writer with that sort is refused with cannot change index sort through both (run by the fact check).

• Who is hit. Callers that expect a substitute value, as the numeric factories document one: IntField.newSortField's missingValue is "a sort value to use for documents with no value in the field" (IntField.java:184). A downstream project's sort DSL lets users pass exactly such a value for string fields, and its tracker (HSEARCH-3254, 2018) notes that it relied on Lucene to throw.
• Default or opt-in. The default codec, IndexWriterConfig and searcher; any index size; assertions on or off. The trigger is the caller's own argument.
• Effect. At search time the argument is silently ignored. As an index sort, the first writer session succeeds and writes segments; only the next IndexWriter, with a fresh config holding the same sort, throws cannot change previous indexSort. In the executed probes no documents were lost, CheckIndex was clean, and with the value removed from the sort the index opens.
• Releases. 10.4.0 to 10.6.0 (the latest release, published 2026-10-29) and main. On 10.4 to 10.6 only callers that moved from the deprecated setter to the constructors are hit; on main the constructors are the only way to set a missing value.
• How likely. Rare: the documented markers, every upstream test and the downstream migrations that were read (Elasticsearch, Solr) pass STRING_FIRST/STRING_LAST.

Two methods, because one method would stop at its first expectThrows. The test shows only that the constructors accept the value; it does not show the search order or the reopen failure. The test does not compile on 10.3.2 or earlier, where the constructors do not exist. To add it to Lucene as the regression test, wrap its 124-character comment line, which google-java-format flags.

The user-visible effects come from a separate print-only probe on the public API (three documents, a KeywordField sort with "zzz", then the same sort as the index sort and a second IndexWriter); its output is in the full output below. It is supporting evidence, not the repro.

A plain main() on the public API with a ByteBuffersDirectory: the search order for "zzz", STRING_FIRST and STRING_LAST, and the index-sort reopen with "zzz", with the value removed, and with STRING_LAST.

package org.apache.lucene.search;

import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.document.KeywordField;
import org.apache.lucene.document.StoredField;
import org.apache.lucene.index.DirectoryReader;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.store.ByteBuffersDirectory;
import org.apache.lucene.store.Directory;

/** Print-only probe for the doc's steps (not the repro): search order and index-sort reopen. */
public class StepsProbe {

  public static void main(String[] args) throws Exception {
    try (Directory dir = new ByteBuffersDirectory()) {
      try (IndexWriter w = new IndexWriter(dir, new IndexWriterConfig())) {
        w.addDocument(doc("apple"));
        w.addDocument(doc("banana"));
        w.addDocument(doc(null));
      }
      try (DirectoryReader r = DirectoryReader.open(dir)) {
        IndexSearcher s = new IndexSearcher(r);
        for (Object mv : new Object[] {"zzz", SortField.STRING_FIRST, SortField.STRING_LAST}) {
          SortField sf = KeywordField.newSortField("tag", false, SortedSetSelector.Type.MIN, mv);
          System.out.println(
              "search missing=" + mv + " sortField=" + sf + " getMissingValue=" + sf.getMissingValue()
                  + " order=" + order(s, sf));
        }
        SortField sv = new SortField("tag", SortField.Type.STRING_VAL, false, "zzz");
        System.out.println("STRING_VAL accepted: " + sv);
      }
    }
    // Within one writer session: two flushed segments merged into one, with the "zzz" index sort.
    Sort zzz = new Sort(KeywordField.newSortField("tag", false, SortedSetSelector.Type.MIN, "zzz"));
    try (Directory dir = new ByteBuffersDirectory()) {
      try (IndexWriter w = new IndexWriter(dir, new IndexWriterConfig().setIndexSort(zzz))) {
        w.addDocument(doc("banana"));
        w.commit();
        w.addDocument(doc(null));
        w.addDocument(doc("apple"));
        w.commit();
        w.forceMerge(1);
        w.commit();
      }
      try (DirectoryReader r = DirectoryReader.open(dir)) {
        StringBuilder sb = new StringBuilder();
        for (int i = 0; i < r.maxDoc(); i++) {
          if (sb.length() > 0) sb.append(", ");
          sb.append(r.storedFields().document(i).get("name"));
        }
        System.out.println(
            "same session, 2 commits + forceMerge(1): OK, segments=" + r.leaves().size()
                + " stored order=" + sb);
      }
    }
    for (Object mv : new Object[] {"zzz", SortField.STRING_LAST}) {
      Sort sort = new Sort(KeywordField.newSortField("tag", false, SortedSetSelector.Type.MIN, mv));
      try (Directory dir = new ByteBuffersDirectory()) {
        try (IndexWriter w = new IndexWriter(dir, new IndexWriterConfig().setIndexSort(sort))) {
          w.addDocument(doc("apple"));
          w.addDocument(doc(null));
        }
        try (DirectoryReader r = DirectoryReader.open(dir)) {
          System.out.println(
              "indexSort missing=" + mv + " config=" + sort + " segment="
                  + r.leaves().get(0).reader().getMetaData().sort());
        }
        try (IndexWriter w = new IndexWriter(dir, new IndexWriterConfig().setIndexSort(sort))) {
          System.out.println("reopen missing=" + mv + ": OK");
        } catch (IllegalArgumentException e) {
          System.out.println("reopen missing=" + mv + ": IllegalArgumentException: " + e.getMessage());
          Sort plain = new Sort(KeywordField.newSortField("tag", false, SortedSetSelector.Type.MIN));
          try (IndexWriter w = new IndexWriter(dir, new IndexWriterConfig().setIndexSort(plain))) {
            System.out.println("reopen with the value removed (" + plain + "): OK");
          }
        }
      }
    }
  }

  private static Document doc(String tag) {
    Document d = new Document();
    d.add(new StoredField("name", tag == null ? "(none)" : tag));
    if (tag != null) {
      d.add(new KeywordField("tag", tag, Field.Store.NO));
    }
    return d;
  }

  private static String order(IndexSearcher s, SortField sf) throws Exception {
    TopDocs td = s.search(MatchAllDocsQuery.INSTANCE, 10, new Sort(sf));
    StringBuilder sb = new StringBuilder();
    for (ScoreDoc sd : td.scoreDocs) {
      if (sb.length() > 0) sb.append(", ");
      sb.append(s.storedFields().document(sd.doc).get("name"));
    }
    return sb.toString();
  }
}

cp $REPRO/TestStringSortMissingValue.java $LUCENE/lucene/core/src/test/org/apache/lucene/search/
cd $LUCENE && ./gradlew :lucene:core:test --tests org.apache.lucene.search.TestStringSortMissingValue -Ptests.seed=DEADBEEF -Ptests.asserts=true
# also with -Ptests.asserts=false

# gradle, main 645d4fe14a20, -Ptests.asserts=true (the same with false, and on the 10.4.0, 10.5.2 and 10.6.0 jars)
TestStringSortMissingValue > testStringValRejectsNonSentinelMissingValue FAILED
    junit.framework.AssertionFailedError: Expected exception IllegalArgumentException but no exception was thrown
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:1812F997AAA093B0]:0)
        ...
        at app//org.apache.lucene.search.TestStringSortMissingValue.testStringValRejectsNonSentinelMissingValue(TestStringSortMissingValue.java:25)
TestStringSortMissingValue > testSortedSetRejectsNonSentinelMissingValue FAILED
    junit.framework.AssertionFailedError: Expected exception IllegalArgumentException but no exception was thrown
        ...
        at app//org.apache.lucene.search.TestStringSortMissingValue.testSortedSetRejectsNonSentinelMissingValue(TestStringSortMissingValue.java:31)
:lucene:core:test (FAILURE): 2 tests, 2  failures

# print-only probe (StepsProbe), main 645d4fe14a20, assertions off
search missing=zzz sortField=<sortedset: "tag"> missingValue=zzz selector=MIN getMissingValue=zzz order=(none), apple, banana
search missing=SortField.STRING_FIRST ... order=(none), apple, banana
search missing=SortField.STRING_LAST ... order=apple, banana, (none)
STRING_VAL accepted: <string_val: "tag"> missingValue=zzz
same session, 2 commits + forceMerge(1): OK, segments=1 stored order=(none), apple, banana
indexSort missing=zzz config=<sortedset: "tag"> missingValue=zzz selector=MIN segment=<sortedset: "tag"> selector=MIN
reopen missing=zzz: IllegalArgumentException: cannot change previous indexSort=<sortedset: "tag"> selector=MIN (from segment=_0(11.0.0):c2) to new indexSort=<sortedset: "tag"> missingValue=zzz selector=MIN
reopen with the value removed (<sortedset: "tag"> selector=MIN): OK
reopen missing=SortField.STRING_LAST: OK

• The contract names two values. STRING_FIRST/STRING_LAST: "Pass this as a missingValue constructor parameter to have missing string values sort first" / "last" (SortField.java:309-324). SortedSetSortField: "Like sorting by string, this also supports sorting missing values as first or last, via SortField#STRING_FIRST or SortField#STRING_LAST" (SortedSetSortField.java:41-42).
• Same check, one type over. #15483 added the rejection for Type.STRING to validateField with TestSort#testStringMissingValues, which asserts "For Type.STRING, missing value must be either STRING_FIRST or STRING_LAST" (TestSort.java:94-104).
• The replaced API. 10.5.2 SortField.setMissingValue threw "For STRING type, missing value must be either STRING_FIRST or STRING_LAST" when type == Type.STRING || type == Type.STRING_VAL; SortedSetSortField.setMissingValue ("Note that this must be STRING_FIRST or STRING_LAST") threw "For SORTED_SET type, ...". The checks date from 2014; executed on 10.4.0, 10.5.2 and 10.6.0 RC1, the setter still rejects "zzz".
• Siblings. BinarySortField: "@param missingValue STRING_FIRST, STRING_LAST, or null" plus a constructor check (BinarySortField.java:72-73, :95-98); ToParentBlockJoinSortField rejects too (ToParentBlockJoinSortField.java:178-198), added after a different #15483 side effect (#15548 / #15727).
• Migration text. "SortField.setMissingValue() has been removed. Missing values should be configured in SortField constructor methods, as they are now final", with a SortedSetSortField example passing SortField.STRING_LAST (MIGRATE.md:258-278). The captured PR metadata and review summaries, the #15483 CHANGES entry and MIGRATE.md announce no widening of the accepted values; #15483's goal was immutability.

The strongest case (the cross-model lenses, the independent reproduction, the intended-behaviour check and the adjudication): the new constructors and KeywordField.newSortField document no @param or @throws for missingValue, so a non-marker value is unsupported input that Lucene never promised to reject; and 10.4 to 10.6 accept it, so a check changes behaviour.

Why it fails: the missing @param is a real gap (fix A adds it), but the only documentation of the parameter names the two markers; the same code, the removed setters and two siblings reject the same input; nothing in the captured PR history suggests the check was dropped on purpose; and "unsupported input" does not explain an IndexWriter that accepts a config once and then refuses the index it produced. The compatibility point bears on where a fix lands (main / 11.x looks lower-risk; the maintainers decide), not on whether the behaviour is intended.

Option A was built and run in a scratch worktree of 645d4fe14a20; it also applies, with line offsets, to upstream main 716d6f216f52. The other options were not built.

The finder's two checks plus the missing @param missingValue javadoc on the three entry points. The Type.STRING message is unchanged, so TestSort#testStringMissingValues still passes.

// SortField.validateField
if (type == Type.STRING || type == Type.STRING_VAL) {
  if (missingValue != null && missingValue != STRING_FIRST && missingValue != STRING_LAST) {
    throw new IllegalArgumentException(
        "For Type." + type + ", missing value must be either STRING_FIRST or STRING_LAST");
  }
}
// SortedSetSortField(String, boolean, SortedSetSelector.Type, Object), after super(...)
if (missingValue != null && missingValue != STRING_FIRST && missingValue != STRING_LAST) {
  throw new IllegalArgumentException(
      "For SortedSetSortField, missing value must be either STRING_FIRST or STRING_LAST");
}

Trade-offs

• Restores the pre-10.4 behaviour and matches Type.STRING, BinarySortField and ToParentBlockJoinSortField.
• Code on 10.4 to 10.6 that passes such a value gets an IllegalArgumentException after upgrading, so main / 11.x looks like the lower-risk target; where it lands and any 10.6.x backport are the maintainers' decision.
• An index already written with such a value has missing-first segments that store no missing value. To keep opening it, the application must rebuild the sort with null as the missing value (run). Keeping the old value, or swapping in STRING_FIRST, is refused: STRING_FIRST sorts the same but is not equal to the stored sort (run by the fact check).

What the checks validated

• Scratch worktree of 645d4fe14a20, gradle, JDK 25. Stock: the test fails 2 of 2. With the fix: the test passes, with assertions on and off.
• With the fix: 13 suites, 193 tests, 0 skipped, 0 failures, with assertions on and off: TestSort, TestSortOptimization, TestSortRandom, TestSortRescorer, TestSortedNumericDocValuesRangeIterator, TestSortedNumericSortField, TestSortedSetSelector, TestSortedSetSortField, TestSearchAfter, TestIndexSorting, TestKeywordField, TestSegmentReordering and the new test.
• :lucene:core:check -x test (google-java-format, ecjLint, forbiddenApis) passes once the test's 124-character comment line is wrapped; google-java-format flags only that line, not the fix.
• The diff applies, with line offsets, to upstream main 716d6f216f52 (not built there). Not run: the full test suite, nightly tests, other modules.

A small static SortField check used by validateField (STRING, STRING_VAL), SortedSetSortField, BinarySortField and ToParentBlockJoinSortField.

Trade-offs

Same behaviour as A with one message, but touches more files and the join module, and changes the existing sibling messages (for example BinarySortField's "missing value for BinarySortField must be null, STRING_FIRST or STRING_LAST"). Not built.

Add @param missingValue ... STRING_FIRST or STRING_LAST and leave the code.

Trade-offs

The silent search order and the late reopen failure stay for callers who miss the doc. Not recommended alone.

Sort documents without a value as if they had the given string, as numeric sorts do.

Trade-offs

Needs a comparator and index-sorter rule for a term that may not exist in the segment, and a new SortedSetSortField serialization. Much larger; the old setters never allowed it.

• First signal (not formal). A planner's migration probe (2026-10-29) compared setMissingValue, deprecated on 10.x and removed on main, with the constructors that replace it.
• Differential check. The finder (a hunt unit) ran 70 cells: SortField(STRING), SortField(STRING_VAL), SortedSetSortField, BinarySortField and ToParentBlockJoinSortField, each with null, STRING_FIRST, STRING_LAST, "zzz", "", a BytesRef and an Integer, forward and reversed. Each cell checked acceptance and the position of the document without a value and, where the field can be an index sort, the stored index sort and a reopen with the same sort. KeywordField.newSortField was not a separate row; it calls the SortedSetSortField constructor (KeywordField.java:188-193), and the steps probe uses it. A second differential compared the setter with the constructor on 10.4.0, 10.5.2, 10.6.0 RC1 and main.
• Reduction. The no-noise gate reduced it to one JUnit class with two methods, each with one expectThrows, one per fix site.