DateTools strings stop sorting by date and stop reading back outside the years 1 to 9999

DateTools (core, org.apache.lucene.document) converts dates to strings for field values and search terms, and its class javadoc says "The strings are structured so that lexicographic sorting orders them by date"; stringToTime converts "a string produced by timeToString or dateToString back to a time". Neither sentence names a range, and timeToString accepts any long.

The year is printed with SimpleDateFormat's yyyy, which is the year of the era, with no era letter, at least four digits. So a BC year prints as the AD year with the same number (1200 BC-06-01 and AD 1200-06-01 both give 12000601; 2 BC-01-01 gives 00020101, sorts after AD 1 and reads back as AD 2-01-01), and a year after 9999 gets a fifth digit (10000-01-01 gives 100000101, sorts between 999 and 1000, and stringToTime throws ParseException on it because the parser is picked by string length). A TermRangeQuery for AD 1000 to AD 1500 matches a 1200 BC document; the classic and flexible query parsers with a date resolution turn date:[1/1/2000 TO 12/31/10000] into an inverted range that matches nothing, and date:[1/1/10000 TO 12/31/10001] into one that matches an AD 1000 document. Dates from 0001-01-01 to 9999-12-31 are unaffected. The code is unchanged since the class was added in 2004; every release from 1.9 to 10.6.0 (10.6.0 by reading) and main is affected. The recommended fix rejects out-of-range times with IllegalArgumentException and documents the range; indexed terms do not change.

• DateTools: a public class in core (org.apache.lucene.document). timeToString(long, Resolution) and dateToString(Date, Resolution) write a date as a string in GMT at a chosen resolution, from YEAR (yyyy, 4 characters) to MILLISECOND (yyyyMMddHHmmssSSS, 17); stringToTime / stringToDate read it back. The class javadoc: "The strings are structured so that lexicographic sorting orders them by date, which makes them suitable for use as field values and search terms" (DateTools.java:28-31). It offers LongPoint as another approach (:37-40).
• Query parser date ranges: with a date resolution set, the classic and flexible query parsers convert the bounds of field:[6/1/2005 TO 6/4/2005] with DateTools (QueryParser.java:50-60).

1. timeToString rounds the time and formats it; it checks no range (DateTools.java:90-93).
2. The format is new SimpleDateFormat("yyyyMMddHHmmssSSS".substring(0, formatLen), Locale.ROOT) (:198). In SimpleDateFormat, y is the year of era and the letter count is a minimum width; there is no era letter G.
3. So 1200 BC prints 1200, exactly like AD 1200, and year 10000 prints 10000, one character more. Compared character by character, 100000101 (10000-01-01) is smaller than 10000101 (1000-01-01).
4. stringToDate parses with TL_FORMATS.get()[dateString.length()] (:117); the array has formats only for lengths 4/6/8/10/12/14/17, so a 9-character string hits null, which is wrapped as ParseException (:118-121). A BC string has a normal length and parses as the AD year with the same number.
5. The classic query parser converts each range bound inside its own catch (Exception e) {} (QueryParserBase.java:454-479); the flexible parser converts both bounds in one try and, in its catch, keeps the texts it has so far (TermRangeQueryNodeProcessor.java:100-142). A bound in year 10000 converts to a 9-character term that sorts among the year-1000 strings. With a later lower bound the range is inverted and matches nothing (date:[1/1/2000 TO 12/31/10000] gives 20000101..100001231). Otherwise it covers the wrong terms: date:[1/1/500 TO 12/31/10000] misses AD 1000, 1200 and 2026 documents, and date:[1/1/10000 TO 12/31/10001] matches an AD 1000 document (executed, classic and flexible). Both parsers read the typed date in the JVM default time zone (QueryParserBase.java:450), so near the edges a typed date can leave the GMT range: under America/Los_Angeles, date:[1/1/2026 TO 12/31/9999] ends in year 10000 GMT and misses a 2026 document.

• Who is exposed. Users of the legacy DateTools encoding (an explicit choice: its javadoc offers LongPoint, and the document package recommends LongField for dates) and of query-parser date ranges with setDateResolution, whose data or queries contain a date outside the years 1 to 9999: historical collections with BC dates; a "never" or "scheduled" date in year 10000 or later, or Long.MAX_VALUE; a bound typed as 12/31/10000.
• What the damage looks like. Range and prefix queries match the wrong documents or none: a 1200 BC document matches AD 1000..1500, a range AD 2000..20000 found neither a 2026 nor a year-10000 document in the devil's-advocate verifier's run, and the parsed date:[1/1/10000 TO 12/31/10001] matches an AD 1000 document. Sorts put year 10000 between 999 and 1000. BC dates read back as the AD date with the same number. The only loud symptom is the ParseException when a year-10000 string is read back.
• Extreme values. Long.MIN_VALUE and Long.MAX_VALUE give strings of 9 to 22 characters that start with 2922; most throw on read-back, and at MINUTE resolution both are silently parsed into a different time (30253281730712). round(Long.MIN_VALUE, YEAR) wraps to a positive time. This edge was confirmed by a verifier's probe only and is not the headline.
• How likely. Rare: present-day data, date pickers and formatters do not produce such dates, and nobody has reported it since 1.9 (2006).
• Scale. One call reproduces it; assertions on and off behave the same; nothing depends on index size, settings or concurrency.
• Not affected. Dates from 0001-01-01 to 9999-12-31 GMT: the control passes at all 7 resolutions, and every day of that range at DAY resolution (3,652,061 days) sorts and reads back exactly; the other resolutions are sampled. Query parsers read typed dates in the JVM default time zone, so a typed date at the very edges can still fall outside the GMT range. Before 1582-10-15 the strings follow the Julian side of GregorianCalendar; they are consistent and in order, so they are not part of this bug. Dates indexed as LongPoint / LongField do not use DateTools.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: 2 BC-01-01 GMT -> 00020101 expected:<-62198928000000> but was:<-62104233600000>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:647)
       org.apache.lucene.document.TestDateToolsBCRoundTrip.testBCDateRoundTrip(TestDateToolsBCRoundTrip.java:35)

• Contract. The ordering sentence has no condition (DateTools.java:28-31). timeToString takes "the date expressed as milliseconds since January 1, 1970, 00:00:00 GMT" (:85), any long, and validates nothing (:90-93). stringToTime / stringToDate convert "a string produced by timeToString or dateToString back to a time" (:96, :108); for year 10000 they throw on that string, for 2 BC they return another time. The @return text says "a string in format yyyyMMddHHmmssSSS or shorter" (:75, :87); a 9-character string has none of that format's prefix lengths (4, 6, 8, 10, 12, 14, 17), which is why stringToDate has no parser for it.
• Tests. TestDateTools uses dates from 1961 to 2005 and 1970; its malformed inputs are 97, 200401011235009999 and aaaa. TestDateSort uses dates in October 2007 and TestCustomSearcherSort dates in February and March 1980. No test covers or pins a year outside 1 to 9999.
• History. The 1.9 CHANGES entry introduced DateTools as the class that "can cope with dates before 1970". Two weeks after adding it, its author answered LUCENE-142 (#1220, historical dates, resolved as Fixed; its patch offered 1970 +/- 5,000 years, BC included) with "The new DateTools class now solves the 'pre-1970' problem". LUCENE-491 (#1569, time zone) and LUCENE-1846 (#2921, locale) fixed earlier ordering breaks; the LUCENE-1846 CHANGES entry says "The usage of the US locale is important to guarantee correct ordering of generated terms", and in that issue uschindler wrote that "the dates in DateTools should sort according to String.compare()".
• Elsewhere in Lucene the era is handled on purpose: the spatial DateRangePrefixTree was built with explicit BC and +-year handling (LUCENE-5648, LUCENE-7278).

The strongest "intended" case (built by the devil's-advocate verifier and the reviews by another model family, all of whom rated it moderate): read literally, @return a string in format yyyyMMddHHmmssSSS is exactly what SimpleDateFormat prints, BC and five-digit years included; the design has been fixed-width with four-digit years and a length-based parser since 2004; the class was written to fix "pre-1970", not BC or year 10000; a four-digit year field is commonly read as 0001 to 9999; the class is legacy (its javadoc offers LongPoint); changing the encoding would change indexed terms; nobody reported it in about 20 years.

Why it fails: no range is stated or checked anywhere (not in DateTools.java, and none of the 34 commits that touched it mentions a year limit, BC or an era; nothing found says BC or five-digit years were considered and left out), and the read-back sentence fails under the literal reading too, because the parser rejects or silently shifts DateTools' own output. Maintainers treat the ordering sentence as a contract (LUCENE-1846), and LUCENE-142 shows historical dates were in scope. Legacy status and term compatibility decide how to fix it (document and validate the range), not whether it is a bug. The adjudication's confidence is medium for exactly this reason: maintainers may well answer "document the range", which is the recommended fix.

Option A is the compatible fix the adjudication, the devil's-advocate verifier and the security screen point to, validated here with gradle. Option B extends the encoding instead; option C only documents the range. Neither was run. With A, replace the issue repro by the fix's own test (TestDateTools.testYearRange), and add a CHANGES entry: callers that pass out-of-range sentinels get an exception from then on.

timeToString (and through it dateToString) checks the time against two constants computed once with Calendar.getInstance(GMT, Locale.ROOT), the calendar round uses: 0001-01-01 00:00:00.000 GMT (-62135769600000) and 9999-12-31 23:59:59.999 GMT (253402300799999). The class javadoc gains a paragraph on the range, and both methods get @throws IllegalArgumentException (+38/-0 in DateTools.java). Checking time is enough: for a time already in the range, round moves it down to the start of its year, month, day, and so on, which stays in the range (outside it this does not hold: round(Long.MIN_VALUE, YEAR) wraps to a positive time).

  public static String timeToString(long time, Resolution resolution) {
    if (time < MIN_TIME || time > MAX_TIME) {
      throw new IllegalArgumentException(
          "time must be between " + MIN_TIME + " (0001-01-01 00:00:00.000 GMT) and " + MAX_TIME
              + " (9999-12-31 23:59:59.999 GMT), got: " + time);
    }
    final Date date = new Date(round(time, resolution));
    return TL_FORMATS.get()[resolution.formatLen].format(date);
  }

Trade-offs

• Every string DateTools can still write has a four-digit AD year, so both javadoc promises hold for all of them, and existing in-range terms are byte-identical: no reindex for indexes that hold only in-range dates.
• A behaviour change: callers that pass Long.MAX_VALUE / Long.MIN_VALUE sentinels or BC dates now get an exception instead of a wrong string. It needs a CHANGES/MIGRATE note; main is the natural target.
• The query parsers stay silent. Each swallows the exception and keeps the unconverted text of the failing bound (QueryParserBase.java:454-479); the flexible parser converts both bounds in one try, so when the lower bound fails the upper one stays unconverted too (TermRangeQueryNodeProcessor.java:100-142). Executed with A for both parsers: date:[1/1/2000 TO 12/31/10000] becomes 20000101..12/31/10000, still inverted, count 0; date:[1/1/10000 TO 12/31/10001] stays raw text and still matches an AD 1000 document, as on stock; date:[1/1/10000 TO 12/31/2030] becomes 1/1/10000..20301231 in the classic parser (3 wrong hits on a 4-document index) and 1/1/10000..12/31/2030 in the flexible one (1 wrong hit). A follow-up could make an out-of-range date fail the parse, and decide what the flexible parser's shared try should keep.
• Existing data. A neither detects nor repairs out-of-range terms already in an index: BC terms stay ambiguous (a BC term is the string of an AD date, so the instant cannot be recovered from it) and year-10000 terms stay unreadable. They need reindexing from the original timestamps; the CHANGES/MIGRATE note should say so.
• Time zones. The range is in GMT, but the query parsers read a typed date in the JVM default time zone: with A, date:[1/1/2026 TO 12/31/9999] under America/Los_Angeles keeps the raw upper text 12/31/9999 (count 0, not 1), and date:[1/1/0001 TO 12/31/2030] under Asia/Tokyo keeps the raw lower text and still misses an AD 1-06-01 document (executed, both parsers).
• Applications with real BC or year-10000 dates get no DateTools encoding for them; they need LongPoint / LongField, which the javadocs already recommend, or their own encoding. stringToDate is unchanged.

What the checks validated

• Scratch worktree at a9779ef3c193, gradle, JDK 25. The repro: on stock 5 of 6 fail (assertions on and off); with A the same 5 fail fast with IllegalArgumentException ("time must be between -62135769600000 ... got: -62198928000000") and the control passes, so no newly written out-of-range term is silently wrong (terms already in an index are untouched).
• The new TestDateTools.testYearRange (the boundaries sort and read back at all 7 resolutions; one millisecond outside, Long.MIN_VALUE and Long.MAX_VALUE throw, for timeToString and dateToString) passes with A and fails on stock; the other 5 TestDateTools tests pass.
• The callers of DateTools: org.apache.lucene.document.*, TestDateSort and TestCustomSearcherSort (61 suites, 502 tests), the whole :lucene:queryparser:test (30 suites, 533 tests) and :lucene:benchmark:test (18 suites, 90 tests): 0 failures with assertions on (seed 5EED0645) and off (5EED0646). The whole :lucene:core:test: 748 suites, 8,890 tests, 0 failures, 322 skipped.
• The core lint run succeeded: checkGoogleJavaFormat, ecjLintMain and renderJavadoc ran and passed; forbiddenApisMain, forbiddenApisTest and ecjLintTest were reported up to date from an earlier successful run in the same worktree, whose log was not kept. Re-run them before the PR.

Keep every string for the years 1 to 9999 byte-identical and add new forms only outside it, for example BC years as - followed by 10000 minus the year (- sorts before every digit, and the complement puts earlier BC years first), and years after 9999 behind a character that sorts after 9.

Trade-offs

Existing indexes stay valid and historical dates become usable, but it is a format extension of a legacy class: new parsers per length, a decision for the query parsers, more tests and maintainer agreement. Solr fixed the same SimpleDateFormat pitfall in its own formatter by moving to java.time (SOLR-8904); that is precedent for calling it a bug, not a sort-preserving design to copy (an ISO year with a minus sign does not sort BC years in date order either).

A javadoc sentence, no behaviour change. Not run.

Trade-offs

Removes the contract mismatch on paper but leaves the silent wrong terms and the ParseException on DateTools' own output.

Method: bounded-exhaustive check on the stock main jar (the study's "semi-formal" tier), a hunt unit (javadoc equivalence claims, enumerated), 2026-10-26, aimed by a code-reading lead that yyyy is the year of era.

• The model ran the stock jar on 142 instants: Long.MIN_VALUE / Long.MAX_VALUE, every year start and end from 4 BC to AD 3, 999/1000, 1581-1583 with the Gregorian cutover days, 1969-1971, 2026, 9998-10001 and 12345, each +-1 ms, at all 7 resolutions, and checked all ordered pairs for order and read-back against the javadoc.
• The finder's log reports violations only outside the years 1 to 9999 and a clean in-range subset. These counts were not re-run independently, so the doc cites only the targeted cases.
• Then an AI finder wrote the public-API JUnit, and independent AI tools reproduced, reviewed, judged and audited it (see the checks).