ReindexingEnrichedDirectoryTaxonomyWriter.reindexWithNewOrdinalData drops categories added since the last commit and reuses their ordinals

ReindexingEnrichedDirectoryTaxonomyWriter (facet module, @lucene.experimental, since 9.10.0) stores extra data in each category's taxonomy document, and to change that data the class javadoc says it is required to call reindexWithNewOrdinalData, whose javadoc promises "The ordinals themselves will be preserved, so the caller does not need to update references to them in the main index". The method records the categories to re-add through new DirectoryTaxonomyReader(Directory), which sees only the last commit, then deleteAll() empties the taxonomy and resets the ordinal counter, only the committed categories are re-added, and the method commits. A category added since the last commit, whose ordinal FacetsConfig.build already stored in main-index documents, is gone, and the next new category gets its ordinal. In the repro Author/Bob (ordinal 3) is dropped, the writer's getSize() goes from 4 to 3, Author/Carol takes ordinal 3, and after both commits the counts are Alice=1 Bob=-1 Carol=2 instead of 1, 1, 1. On a taxonomy that was never committed the call throws IndexNotFoundException. Every release from 9.10.0 to 10.6.0 (10.6.0 by reading) and main is affected; reading the writer's own near-real-time view, new DirectoryTaxonomyReader(this), fixes it.

• Taxonomy and ordinal. Taxonomy facets keep each label, such as Author/Bob, once in a side index; its document number there is the label's ordinal. FacetsConfig.build asks the taxonomy writer for the ordinal (addCategory) and stores it in the main-index document as a doc-values field; facet counts are kept per ordinal. Ordinals start at 0 (the root) and grow by one per new label; the writer's getSize() is the number of ordinals. A label, once added, is never deleted.
• Ordinal data. ReindexingEnrichedDirectoryTaxonomyWriter adds fields of your choice to each label's taxonomy document. To change them you call reindexWithNewOrdinalData(appender): it deletes the taxonomy, re-adds every label in ordinal order with the new data, and commits, so the main index needs no update.
• Commit point vs. the writer's view. new DirectoryTaxonomyReader(Directory) opens the last commit; new DirectoryTaxonomyReader(taxoWriter) opens the writer's near-real-time view, which includes labels added since the last commit.

1. reindexWithNewOrdinalData (ReindexingEnrichedDirectoryTaxonomyWriter.java:87-104) calls recordPathsInOrder(getDirectory()) (:91, :94), which opens new DirectoryTaxonomyReader(d) (:63): the last commit only (DirectoryTaxonomyReader.java:128-129, :228-230).
2. deleteAll() (:97; DirectoryTaxonomyWriter.java:902-919) deletes every taxonomy document, including the labels added since that commit, and resets the ordinal counter to 0 (:907).
3. Only the recorded labels are re-added, in ordinal order (:100-102), and commit() (:103) persists the shorter taxonomy.
4. The next new label gets the next free ordinal (DirectoryTaxonomyWriter.java:469), which is the first dropped one. Main-index documents that hold it are counted and labelled under the new label. Until a new label takes it, the ordinal lies past getSize() and faceting throws ArrayIndexOutOfBoundsException (seen with commit-point readers and with a near-real-time pair, in the devil's advocate's and the diagram probes).
5. On a taxonomy that was never committed there is no commit to open: :63 throws IndexNotFoundException before anything is deleted.

This doc's test (default FacetsConfig; the test framework's randomized directory and main-index settings): commit after Author/Alice (ordinals 0-2); a document with Author/Bob gets ordinal 3 (getSize() 4); reindexWithNewOrdinalData; getSize() is now 3 and getOrdinal(Author/Bob) is -1; a document with Author/Carol gets ordinal 3; after both commits FastTaxonomyFacetCounts gives Alice=1 Bob=-1 Carol=2. With a commit right before the call, all ordinals are kept. Same on main, 10.5.2 and 9.10.0, assertions on and off, and for every seed tried; the finder's and the independent reproducer's tests and the diagram probe, with a plain new IndexWriterConfig(...), fail the same way.

• Who is exposed. Users of this opt-in, @lucene.experimental class (no special configuration otherwise) who call reindexWithNewOrdinalData while the writer holds a category added since its last taxonomy commit: a writer that keeps indexing between commits (for example a scheduled refresh of per-category data), or a batch build that adds its last categories and reindexes without committing first. A read-only code search found no use of the class in Elasticsearch, OpenSearch or Solr; custom applications that embed the facet module are the ones exposed.
• Trigger. Deterministic and single-threaded; one uncommitted category is enough; no special values, timing or scale.
• Effect. Silent and persisted: the method's own commit writes a taxonomy without those categories, main-index documents keep the old ordinals, and the next new categories take them, so those documents are counted and labelled under other categories, also from freshly opened readers. Before a new category takes the ordinal, faceting over the affected documents throws ArrayIndexOutOfBoundsException (seen with commit-point readers and with a near-real-time pair). A later correct reindex keeps the wrong mapping, so repair needs the affected main-index documents re-indexed (inferred, not run).
• Workaround. Commit the taxonomy, with indexing paused, right before the call. Pausing matters: other threads can add categories between a caller's commit() and the call.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected:<2> but was:<-1>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:647)
       org.junit.Assert.assertEquals(Assert.java:633)
       org.apache.lucene.facet.taxonomy.directory.TestReindexDropsUncommittedCategories.testUncommittedCategoryOrdinalIsNotPreserved(TestReindexDropsUncommittedCategories.java:36)

• Contract. ReindexingEnrichedDirectoryTaxonomyWriter.java:82-86: "Delete the existing taxonomy index and recreate it using new ordinal data. The ordinals themselves will be preserved, so the caller does not need to update references to them in the main index." The class javadoc (:31-34) says that to update the custom data "it is required to" call this method. The deleteAll javadoc (DirectoryTaxonomyWriter.java:893-898) names it as the example of regenerating the taxonomy "taking care that ordinals are indexed in the same order as before".
• The writer contradicts itself. TaxonomyWriter.getSize: "the taxonomy contains ordinals 0 through getSize()-1" (TaxonomyWriter.java:82-93); the same writer reports 4, then 3. TaxonomyReader: "once a category is added, it can never be deleted" (TaxonomyReader.java:33-35). These general rules weigh less than the method's own promise, since replaceTaxonomy exists and the method itself starts with "Delete the existing taxonomy index and recreate it"; the strongest contract is that the ordinals are preserved.
• Tests. The only upstream test, TestOrdinalData.testReindex (:169-181), runs after its setup committed the taxonomy (:110) and checks the new data, not ordinals.
• History. In the review of #12337, which added the class, mikemccand asked for deleteAll warnings: "you must carefully regenerate the ordinals in the right order?" That puts the burden on this method. The commit-point reader was there from the first revision with the class and was never discussed.
• Legal trigger. Public API only, in the documented taxonomy-first commit order (TaxonomyWriter.java:36-45).

Strongest case (the devil's advocate; the cross-model intended-lens reviewer called it weak, the later cross-model fact check moderate, since several design signals point to a quiesced rebuild at a commit boundary): TaxonomyWriter says "What the writer writes is only available to (new) readers after the writer's commit() is called", so "the existing taxonomy index" could mean the committed one; the only test commits first; the design thread before #12336 listed the option "New ordinal data can only come in when a new taxonomy gets built"; the class is @lucene.experimental ("so we reserve the right to break them"); deleteAll() bumps the taxonomy epoch, so an NRT SearcherTaxonomyManager treats a reindex like replaceTaxonomy; and in LUCENE-4886 (2013, Not A Problem) a maintainer told a user to commit before opening a Directory-based taxonomy reader.

Why it fails: the writer's own getSize() shrinks and its addCategory reuses ordinal 3 for another label, so from the writer's side the existing taxonomy includes Bob; the visibility rule describes readers, not a precondition on a synchronized writer method that runs on the live writer and commits by itself; no javadoc, CHANGES entry or test states a precondition; the design option is not the API that shipped, and even a batch build loses its last categories if it reindexes before committing; experimental permits API changes, not silently losing categories that committed documents reference; the epoch behaviour is separate and unchanged by the fix; in LUCENE-4886 the user's own code opened the reader, here Lucene's method does, and the fix follows that thread's other advice (open the reader on the writer). This is a claim about the documented contract; whether the author had only quiesced, just-committed rebuilds in mind is not known, and a maintainer may answer by documenting that (option C). Verdict: a defect under the documented API. Expect the reply "commit first"; answer with the javadoc promise and the getSize() 4-to-3 test.

Option A was built and validated with gradle in a scratch worktree, the whole facet module included; option B was run on the new test and the two closest suites; options C and D were not built.

One file, ReindexingEnrichedDirectoryTaxonomyWriter.java: recordPathsInOrder opens new DirectoryTaxonomyReader(this) instead of new DirectoryTaxonomyReader(d), and reindexWithNewOrdinalData no longer passes the Directory. It is the finder's fix and the idiom a maintainer recommended in LUCENE-4886.

private List<FacetLabel> recordPathsInOrder() throws IOException {
  List<FacetLabel> paths = new ArrayList<>();

  // Read this writer's own (NRT) view, which includes categories added since the last commit.
  DirectoryTaxonomyReader taxoReader = new DirectoryTaxonomyReader(this);
  ...

// reindexWithNewOrdinalData: drop "Directory d = getDirectory();"
List<FacetLabel> ordinalToPath = recordPathsInOrder();

Trade-offs

• Safe because the taxonomy index has no deletions apart from deleteAll() and its merge policy keeps doc ids ordered, so the near-real-time reader's doc ids are the ordinals, as the writer's own lookups already assume; on the success path the reader is closed before deleteAll() (there is no try/finally, as on stock); the method holds the writer's monitor, so no new category can be added while it runs.
• Cost: opening the near-real-time reader forces a full flush of the taxonomy's internal writer: still-buffered documents are written to segment files, with no commit point and no commit fsync. Not benchmarked; the method rewrites the whole taxonomy anyway.
• Taxonomies already damaged are not repaired; the affected main-index documents need re-indexing.
• The file is identical from 9.10.0 to main, so the patch applies to every branch; open PR #16847 edits lines 68-70 of the same method and may need a trivial rebase.

What the checks validated

• Scratch worktree at a9779ef3c193, gradle, JDK 25: the new test fails 3 of 4 on stock and passes 4 of 4 with the fix, assertions on and off; with TestOrdinalData and TestDirectoryTaxonomyWriter: 22 tests, 0 failures, 1 skipped (testHugeLabel).
• The whole facet module: 40 suites, 310 tests (306 upstream + 4 new), 0 failures, with assertions on (7 skipped) and off (8 skipped); :lucene:facet:check -x test (google-java-format, ecjLint, forbiddenApis) passes.
• With the patched class shadowed in the standalone runner: the new test, the finder's 2 tests, the independent reproducer's 4 tests, the devil's advocate's 3 probes and the security probe pass. In the finder's TLA+ model, recording the writer's view is clean for 3 labels and 4 documents (1,562 distinct states).

Call commit() before recordPathsInOrder(d) (+3 lines).

Trade-offs

Correct because the method is synchronized like addCategory's add path. Costs an extra commit() call: when changes are pending (the case that triggers the bug) it writes and syncs an intermediate commit point, and with nothing pending it commits nothing (IndexWriter.commit() returns -1); the uncommitted categories become durable before the caller's next main-index commit, which the taxonomy-first rule allows and the method's final commit does anyway.

What the checks validated

Gradle on main: the new test passes 4 of 4; with TestOrdinalData and TestDirectoryTaxonomyWriter, 22 tests pass (assertions on); format check passes. The whole module was not run.

"Commit the taxonomy, with indexing paused, before calling", and throw IllegalStateException when the writer holds uncommitted categories.

Trade-offs

Fails loudly instead of corrupting, but moves the burden to every caller and narrows an unconditional promise. A maintainer may prefer it, citing LUCENE-4886. Not built.

Document that categories added since the last commit are dropped.

Trade-offs

No code risk, but nothing stops the silent damage. Not built.

Method: TLA+ model checking (TLC) (semi-formal tier: a hypothesis formed while modelling, then witnessed by the tool), in a hunt unit on the taxonomy and main-index pair.

• The model TaxoReindex.tla keeps the writer's taxonomy, its last commit and main-index documents that store ordinals; the reindex is one atomic step (the method is synchronized) that snapshots either the commit (as coded) or the writer's view (the fix). Its invariant OrdinalsPreserved: every main-index document still resolves to the label it was indexed with.
• With the commit snapshot, TLC reports a 4-state trace, AddCat -> AddDoc -> Reindex, that violates OrdinalsPreserved (19 distinct states). With the writer's view the model is clean for 3 labels and 4 documents (1,562 distinct states), and a probe invariant confirms that reindexes after documents are explored.
• The trace was reduced to a public-API JUnit on the stock jars, then checked by an independent reproducer, a devil's advocate, reviewers from another model family (OpenAI GPT-5.6, reading only), an adjudication and an audit.