SearcherTaxonomyManager's IndexWriter constructor opens the taxonomy before the index, so its first pair can be mismatched

SearcherTaxonomyManager's IndexWriter (near-real-time) constructors open new DirectoryTaxonomyReader(taxoWriter) before DirectoryReader.open(writer, applyAllDeletes, false). A document indexed by another thread in between, together with a new category, is visible to the first searcher while the paired taxonomy reader does not know its category. getOrdinal returns -1 for it, FastTaxonomyFacetCounts over the pair throws ArrayIndexOutOfBoundsException when the counted hits include that document, and maybeRefresh() keeps the pair until the index changes again. TaxonomyReader's javadoc and the manager's own refreshIfNeeded both open the index first for exactly this reason. Query-time only, no index damage; it needs the manager to be built while new categories are being indexed. Present since the class was added in 4.3.0; executed failures on 15 releases from 4.3.0 to 10.6.0 and on main. Swapping the two opens fixes it.

1. Indexing a faceted document adds its new categories to the taxonomy writer (FacetsConfig.build) and then the document to the main IndexWriter, from any thread; IndexWriter is "completely thread safe" (IndexWriter.java:167-168).
2. The constructor body, shared by all three IndexWriter constructors (SearcherTaxonomyManager.java:57-103): new DirectoryTaxonomyReader(taxoWriter) takes the taxonomy snapshot first (:93); then DirectoryReader.open(writer, applyAllDeletes, false) (:96-97), which calls writer.getReader(...) (DirectoryReader.java:141-144), takes the index snapshot.
3. The window. A document whose addDocument completes between the two snapshots is in the searcher, but its new categories were added after the taxonomy snapshot, so getOrdinal returns INVALID_ORDINAL (-1) for them.
4. Counting fails when the counted hits include such a document. With dense counting (always used for a taxonomy of fewer than 1024 ordinals, TaxonomyFacets.java:104) FastTaxonomyFacetCounts counts in its constructor into an array sized taxoReader.getSize() (TaxonomyFacets.java:134), and counts[(int) ordinal]++ (FastTaxonomyFacetCounts.java:93) throws for the unknown ordinal. With sparse counting (1024 or more ordinals and hits under a tenth of maxDoc, TaxonomyFacets.java:103-122) the counting itself succeeds (IntIntHashMap.addTo, FastTaxonomyFacetCounts.java:97, :110), and getTopChildren throws when it looks up the parent of the unknown ordinal (TaxonomyFacets.java:482; Index 1104 out of bounds for length 1104 with 1,104 categories). By code reading, getPath of an unknown ordinal throws IllegalArgumentException. A facet count whose hits exclude the document succeeds on the same pair.
5. Refresh keeps it. refreshIfNeeded opens the index first (SearcherTaxonomyManager.java:195) and returns null when nothing changed (:196-197), so maybeRefresh() keeps the bad pair. The first refresh after the next index change opens the index, then the taxonomy (:204), and gives a matched pair.

For ordinals the stale taxonomy reader does know, the labels are right, because categories are append-only; drill-down terms are built without a taxonomy reader. This is an argument from the code, not an exhaustive test. Searches that do not count facets do not read the taxonomy reader, and a DrillDownQuery builds its terms without one (DrillDownQuery.java:50-51, :116-119); taxonomy-backed DrillSideways does take the taxonomy reader and counts with it through FastTaxonomyFacetCounts (DrillSideways.java:89-90, :157-166), so its facet counts fail the same way (code reading, not run). Only readers are opened, so the index is not damaged.

Claimed for the IndexWriter constructors only. The Directory constructor has the same order, but its javadoc NOTE already says it can give an "unsync'd" pair unless commit and maybeRefresh() run in the same thread (SearcherTaxonomyManager.java:105-111). The constructors that take already opened readers use the caller's pair.

• Who. Taxonomy facets (an opt-in module) with SearcherTaxonomyManager(IndexWriter, ...), otherwise default IndexWriterConfig, codec and searcher. The manager must be built while another thread indexes documents that add new categories: indexing that starts before the search side, a manager created lazily, or one re-created after DirectoryTaxonomyWriter.replaceTaxonomy, which the class javadoc requires ("you must open a new SearcherTaxonomyManager afterwards", SearcherTaxonomyManager.java:37-38).
• Effect. Facet counts on the first pair whose hits include such a document fail with ArrayIndexOutOfBoundsException (counts that exclude it succeed), and a new category reads as absent (getOrdinal = -1). Under continuous indexing it lasts about one refresh interval; if the index then stays unchanged, it stays.
• How likely: rare. Applications that build the manager before indexing starts, as upstream's testNRT does, cannot hit it. When construction does overlap indexing of new categories, the window is hit often: 62 to 178 of 300 managers in hook-free stress runs where every document adds a new category (a worst case). No public report of this constructor-order defect was found.
• Downstream. A read-only code search found no use of SearcherTaxonomyManager or DirectoryTaxonomyReader in Elasticsearch, OpenSearch or Solr. The Lucene.NET port has the same constructor order.

The hook fires on "flush at getReader", which IndexWriter.getReader logs right after ensureOpen() and before it takes any lock or flushes (IndexWriter.java:516-526); the constructor reaches it through DirectoryReader.open(writer, ...) (SearcherTaxonomyManager.java:97), after it opened the taxonomy reader (:93). Indexing there, on the constructing thread, once, creates the ordering a second indexing thread creates when its addDocument completes at that moment; it is a re-entrant call, not ordinary scheduling. Hook-free runs (one thread indexing, managers built in a loop, not part of this repro) show that ordinary scheduling reaches it too: 62 to 178 of 300 managers got a mismatched first pair.

The first assertion (the searcher sees the document, numDocs 1) passes; the second (its category is known) fails. The facet-counting exception follows from the -1 ordinal and is not asserted; the hunt's longer test shows it (Index 3 out of bounds for length 3, below).

Gradle chooses tests.asserts from the seed when the flag is not given (DEADBEEF chooses off); the test fails either way. Run ./gradlew tidy before putting the test in a PR: checkGoogleJavaFormat wants the IndexWriter writer = ... line split over two lines.

Regression test: this same test; it passes with the recommended fix.

cp $REPRO/TestSearcherTaxonomyManagerCtorOrder.java $LUCENE/lucene/facet/src/test/org/apache/lucene/facet/taxonomy/
cd $LUCENE && ./gradlew :lucene:facet:test --tests TestSearcherTaxonomyManagerCtorOrder -Ptests.seed=DEADBEEF -Ptests.asserts=true
# assertions off fails the same way: -Ptests.asserts=false

# gradle, scratch worktree of 645d4fe14a20, -Ptests.seed=DEADBEEF -Ptests.asserts=true
TestSearcherTaxonomyManagerCtorOrder > testFirstPairKnowsTheCategoryOfItsDocument FAILED
    java.lang.AssertionError: taxonomy ordinal of Author/Bob. Actual: -1
        at __randomizedtesting.SeedInfo.seed([DEADBEEF:35F52105B867264B]:0)
        at junit@4.13.2/org.junit.Assert.fail(Assert.java:89)
        at junit@4.13.2/org.junit.Assert.failEquals(Assert.java:187)
        at junit@4.13.2/org.junit.Assert.assertNotEquals(Assert.java:201)
        at org.apache.lucene.facet.taxonomy.TestSearcherTaxonomyManagerCtorOrder.testFirstPairKnowsTheCategoryOfItsDocument(TestSearcherTaxonomyManagerCtorOrder.java:75)
:lucene:facet:test (FAILURE): 1 test, 1  failure
# the same with -Ptests.asserts=false; the same on the 10.6.0 and 10.5.2 release jars

# the hunt's longer test (the scenario of the steps above), main, seed DEADBEEF; the pair is acquire()d again after maybeRefresh()
first pair: searcher numDocs=2, taxonomy size=3, taxonomy ordinal of Author/Bob=-1;
  counts: [facet counting threw java.lang.ArrayIndexOutOfBoundsException: Index 3 out of bounds for length 3];
  after maybeRefresh() (returned true): [facet counting threw java.lang.ArrayIndexOutOfBoundsException: Index 3 out of bounds for length 3]
  expected:<[Alice=1 Bob=1]>

# with the fix: :lucene:facet:test (SUCCESS): 1 test, assertions on and off

• Contracts. TaxonomyReader: "it is important to open the taxonomy after opening the main index ... This order is important, otherwise it would be possible for the main index to refer to a category which is not yet visible in the old snapshot of the taxonomy" (TaxonomyReader.java:46-55). SearcherAndTaxonomy: "Holds a matched pair of IndexSearcher and TaxonomyReader", "Matching point-in-time DirectoryTaxonomyReader" (SearcherTaxonomyManager.java:43-50).
• No caveat on these constructors. "Creates near-real-time searcher and taxonomy reader from the corresponding writers" (:57); the 3-argument one is not marked Expert. The only "unsync'd pair" NOTE is on the Directory constructor (:105-111). SearcherTaxonomyManager is not @lucene.experimental (TaxonomyReader is).
• The code's own intent. refreshIfNeeded: "Must re-open searcher first, otherwise we may get a new reader that references ords not yet known to the taxonomy reader" (:189-191) and "taxonomy should always be ahead of searchers. Otherwise, searchers might reference ordinals that taxonomy doesn't know of" (:200-202). The first comment (:189-191) came in the commit that created the class (03694e76930, 2013), together with the constructor's opposite order; the second (:200-202) was added by f3796ecf22b6 (#14443, 2025).
• Maintainers on record. In #6203 (LUCENE-5139, Won't Fix) an application refreshed its two readers separately (sm.maybeRefresh(), then the taxonomy), and its search threads acquired them separately. A maintainer (shaie) suggested reopening the taxonomy before sm.maybeRefresh() there, because "It's usually ok if taxoReader sees more ordinals than IndexSearcher, not vice versa" ("usually": not after replaceTaxonomy with deleteAll()), and recommended SearcherTaxonomyManager. That advice is about publishing two readers separately, not about the snapshot order inside one pair, where TaxonomyReader's javadoc requires the index first. In a 2014 java-user thread they said the manager returns "a sync'd IndexSearcher and DirectoryTaxonomyReader".
• Tests. None asserts or relies on the taxonomy-first order; TestSearcherTaxonomyManager.testNRT builds the manager (:160) before it starts its indexer thread (:192), so it never exercises the window.

The strongest case: applications normally build the manager once before indexing starts, as testNRT does; the Directory constructor already warns about unsync'd pairs; the taxonomy APIs are experimental; and LUCENE-5139 put pairing on the application.

Why it fails: no javadoc states a "build before indexing" precondition, IndexWriter is thread safe, and the class javadoc itself requires building a new manager at runtime after replaceTaxonomy; the NOTE is attached only to the Directory constructor; in LUCENE-5139 the application refreshed and published its two readers separately, while here the manager builds one pair itself, and the module's own javadoc and refresh code prescribe the opposite order; no comment, test or issue defends the constructor's order. Cross-model reviews (OpenAI GPT-5.6) rated the works-as-designed case weak.

Option A was built and run; B is a workaround for users today; C is not recommended.

In the IndexWriter constructor body (SearcherTaxonomyManager.java:93-98) open DirectoryReader.open(writer, applyAllDeletes, false) first and new DirectoryTaxonomyReader(taxoWriter) second; the same swap in the Directory constructor (:118-122) is optional, a consistency change: it does not remove that constructor's documented commit/refresh limitation (:105-111). A taxonomy snapshot taken after the index snapshot contains every category the index snapshot's documents use, because categories are added before their document and never removed.

DirectoryReader reader = DirectoryReader.open(writer, applyAllDeletes, false);
DirectoryTaxonomyReader taxoReader;
try {
  taxoReader = new DirectoryTaxonomyReader(taxoWriter);
} catch (Throwable t) {
  IOUtils.closeWhileSuppressingExceptions(t, reader);
  throw t;
}
current =
    new SearcherAndTaxonomy(
        SearcherManager.getSearcher(searcherFactory, reader, null), taxoReader);

Trade-offs

• Same cost as today: the same two readers are opened, only in the other order.
• It gives the one-sided invariant the javadoc asks for (the taxonomy reader may be ahead of the index reader, never behind), not an atomic snapshot of both indexes; the class still requires a new manager after replaceTaxonomy.
• Like the current code, it still leaks the taxonomy reader if the SearcherFactory throws (SearcherManager.getSearcher releases the index reader, SearcherManager.java:262-264); that is a separate bug (registry LFV-309), not part of this fix.
• The diff is the hunt's fix with two lines wrapped for google-java-format.

What the checks validated

• This doc's runs: scratch worktree of 645d4fe14a20 with the minimal test copied in, gradle, seed DEADBEEF. Before the fix the test failed with assertions on and off; with the diff, :lucene:facet:test (SUCCESS): 1 test with assertions on and off.
• Re-run on 2026-10-30 in a fresh scratch worktree: the stock test fails (ordinal -1); with the diff, TestSearcherTaxonomyManager* (the minimal test plus TestSearcherTaxonomyManager's 6 tests) gives SUCCESS: 7 tests, 1 skipped (the @Nightly testNRT) with assertions on and off.
• The whole facet module with the fix, assertions on: 40 suites, 307 tests, 7 skipped, 0 failures. TestSearcherTaxonomyManager with -Ptests.nightly=true, which adds the @Nightly testNRT (concurrent indexing and refresh through the IndexWriter constructor): 6 of 6 pass.
• ecjLint and forbiddenApis (main and test) pass; checkGoogleJavaFormat flags only one line of the test file, not the fix.
• Earlier (hunt, the same swap): every repro passes, the hook-free plain-threads program gives 0 of 300 mismatched pairs, upstream TestSearcherTaxonomyManager 5 of 5, and the same swap on the 4.3.0 source passes there.
• Not run: the full Lucene suite, benchmarks (the fix opens the same two readers, in the other order).

Open DirectoryReader.open(writer) first, then new DirectoryTaxonomyReader(taxoWriter), and pass both to the constructor that takes already opened readers (SearcherTaxonomyManager.java:127-135).

Trade-offs

Loses the replaceTaxonomy epoch check, since that constructor keeps no taxonomy writer (:154-155). Not run as a separate test.

Add a NOTE like the Directory constructor's to the IndexWriter constructors.

Trade-offs

Contradicts the class javadoc's instruction to re-create the manager after replaceTaxonomy, and leaves the exception in place.

Method: TLA+ model checking (TLC), then an executed public-API test (the hunt classes it as semi-formal: the order was noticed while modelling, and the tool then produced the witness).

• Model. TaxoPair.tla models the two writers, indexing (add a category, then a document), the constructor's two opens and the refresh. With the constructor as coded, TLC violates PairSafe (every ordinal the searcher's documents use is known to the paired taxonomy reader) in 5 states: Ctor1 (taxonomy snapshot), AddCat, AddDoc, Ctor2 (index snapshot). With the index-first order it is clean (28,053 and 3,172,539 distinct states in two sizes); with no indexing during construction it is clean too; a refresh-order mutant violates it, so the property is not vacuous.
• Execution. A JUnit on the public API paused the constructor at getReader's first InfoStream message while another thread added a document with a new category; it failed on stock main with assertions on and off, 20 of 20 iterations, and passed with the swap. The minimal repro above is that test cut down to one thread and one claim, checked by two assertions (the searcher sees the document, and its category is known).