Flexible query parser: a processor put on the pipeline through listIterator(), subList() or replaceAll() gets no config

QueryNodeProcessorPipeline, the processor chain of the flexible query parser, implements java.util.List and promises to set its QueryConfigHandler "on all processor on pipeline". Its own add, add(int), addAll, addAll(int) and set do that for each processor they put on the pipeline. But listIterator(), listIterator(int) and subList() return the views of its private ArrayList, and replaceAll is not overridden, so a processor put on the pipeline through ListIterator.add/set, subList(..).add/set, List.replaceAll, Collections.replaceAll on a pipeline of 11 or more processors, or Collections.copy of 10 or more processors into it (JDK 25's thresholds) is not given the pipeline's handler: a fresh processor keeps null. Nothing hands it out again before parsing.

Four stock processors check for a null handler and silently do nothing: on StandardQueryParser with n configured as an int point field, a PointQueryNodeProcessor swapped in through listIterator().set leaves n:5 a TermQuery that finds 0 of the 1 matching document, a swapped BoostQueryNodeProcessor drops field boosts, and the point-range and default-slop processors lose their work too. Nine others throw NullPointerException at parse. It is opt-in (only code that edits a pipeline this way), present since 3.1.0, and the workaround is to call setQueryConfigHandler again after editing. The fix wraps the two views so their add and set hand out the handler.

• Pipeline: QueryNodeProcessorPipeline implements QueryNodeProcessor, List<QueryNodeProcessor> (QueryNodeProcessorPipeline.java:39). It keeps the processors in a private ArrayList (:41) and runs them in order (process, :75-82). It is not RandomAccess.
• Config: the QueryConfigHandler. Class javadoc: "When a QueryConfigHandler object is set on a QueryNodeProcessorPipeline, it also takes care of setting this QueryConfigHandler on all processor on pipeline" (:36-37). setQueryConfigHandler loops over the processors (:93-99).
• Views: java.util.List hands out a ListIterator and a subList view that can add and replace elements; the List default replaceAll and the Collections helpers are built on them.

1. Every method the pipeline overrides to put a processor on it also hands it the config: add (:105-113), add(int) (:119-122), addAll (:128-136), addAll(int) (:142-150), set (:268-276): processor.setQueryConfigHandler(this.queryConfig).
2. listIterator() (:220-222), listIterator(int) (:228-230) and subList (:290-292) return the private ArrayList's own views. Their add and set write straight into the ArrayList, so the processor keeps the config it had: none, for a fresh one.
3. The pipeline does not override replaceAll; JDK 25's default is ListIterator li = this.listIterator(); while (li.hasNext()) li.set(operator.apply(li.next())); (List.java:437-443). Collections.replaceAll uses the list iterator on a list that is not RandomAccess and has 11 or more elements, Collections.copy when it copies 10 or more elements (Collections.java:111-112, :587, :870); StandardQueryParser's pipeline has 21. These thresholds are JDK 25 implementation choices, not part of the List contract; another JDK may differ. reversed() inherits the gap through its list iterator and replaceAll; reversed().set calls the pipeline's set and is fine.
4. Nothing repairs it before the parse: QueryParserHelper.parse just calls process (QueryParserHelper.java:213-223). Only setQueryConfigHandler (:185-192), setQueryNodeProcessor (:100-103) and the helper's constructor (:71-73) hand the config out. Calling a setter such as setPointsConfigMap again does not help: it only writes into the config.
5. PointQueryNodeProcessor (:65-67), PointRangeQueryNodeProcessor (:61-63), BoostQueryNodeProcessor (:47-49) and DefaultPhraseSlopQueryNodeProcessor (:51-53) check if (config != null) and otherwise do nothing. Others dereference it, for example AnalyzerQueryNodeProcessor.process (:82), and throw NullPointerException.

StandardQueryParser with n as an int point field and field boost 2 on f; one document with IntPoint n = 5. Each of the 21 stock processors was also replaced through 7 paths and checked on 22 queries, against the same edit through set(k, p) or add(k, p).

Per view path, 4 processors go silent (point, point range, default slop, boost), 9 throw NullPointerException (wildcard, fuzzy, regexp, term range, leading wildcard, analyzer, boolean modifier, multi-term rewrite, interval; 4 of them on every query, 5 only on some query kinds) and 8 give the same result on the 22 queries. In total 147 cases: 69 equal, 24 silent, 54 loud. The list contents are always right; only the config is missing.

• Who is exposed. Code that customises a flexible query parser pipeline through these paths: editing StandardQueryParser's pipeline in place after a cast of getQueryNodeProcessor() (the in-tree idiom in TestQPHelper.java:163-167, and the advice in java-user threads of 2010 and 2014, which use add), or a pipeline built with new QueryNodeProcessorPipeline(config) and used directly (like TestSpanQueryParserSimpleSample.java:98-103, which uses add). The package docs invite this ("select your favorite query processors and build a pipeline", standard/package-info.java:55-58), and LUCENE-1937 added the List API "if you want to extend or modify an existing pipeline".
• Natural idioms reach it. "Insert my processor after X" is the ListIterator.add loop; "swap X for my subclass" is replaceAll(p -> p instanceof X ? new MyX() : p) or Collections.replaceAll(pipeline, old, new MyX()).
• Not affected. An unedited parser; edits through the pipeline's add, add(int), addAll, set, addFirst/addLast; moves and removals (Collections.swap, reverse, rotate, shuffle, sort, remove); Collections.replaceAll on 10 or fewer processors; a pipeline edited before it is installed with setQueryNodeProcessor or the parser helper's constructor, which hand the config to every processor.
• Loud vs silent. In the stock differential (each of the 21 stock processors swapped through the 6 view paths, 22 queries), 9 processors give a NullPointerException on the first parse that reaches them and 8 give the same result. Swapping the other 4, the point, point-range, boost or default-slop processor, or inserting a custom processor that tolerates a missing config, gives silently wrong queries: point queries that match nothing, lost boosts (ranking changes), lost default slop (fewer phrase matches).
• Scale. None: one edit and one query reproduce it, with assertions on or off.
• Workaround. After editing, call qp.setQueryConfigHandler(qp.getQueryConfigHandler()), pipeline.setQueryConfigHandler(...) or qp.setQueryNodeProcessor(pipeline), or edit with add(k, p) / set(k, p).

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected a PointRangeQuery, got org.apache.lucene.search.TermQuery n:5
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.assertTrue(Assert.java:42)
       org.apache.lucene.queryparser.flexible.core.processors.TestPipelineListIteratorSetConfigHandler.testProcessorSetThroughListIteratorGetsConfigHandler(TestPipelineListIteratorSetConfigHandler.java:42)

• The class states the rule and keeps it everywhere else. The javadoc promises the pipeline "takes care of setting this QueryConfigHandler on all processor on pipeline" (QueryNodeProcessorPipeline.java:36-37). Each of the five overriding mutators that put a processor on it calls processor.setQueryConfigHandler(this.queryConfig) (:105-150, :268-276), and so did the original single entry point, addProcessor, before LUCENE-1937. The views are the only way in that skips it; no exemption for them was found in the checked source and javadocs, tests, CHANGES/MIGRATE or prior-art searches.
• Same operation, different result. java.util treats add(k, p) and listIterator(k).add(p), or set(k, p) and listIterator().set(p), as the same element operation; here they parse differently. Collections.replaceAll sets the config on a pipeline of 10 processors and not on one of 11.
• The API exists for this edit. #3012 (LUCENE-1937): the pipeline was "not very flexible when the user wants to extend/modify an already created pipeline", and the proposed method was insertAfter(processor, newProcessor), the ListIterator.add idiom. CHANGES 3.1: "QueryNodeProcessorPipeline now implements the List interface, this is useful if you want to extend or modify an existing pipeline."
• Tests. No queryparser test uses listIterator, subList or replaceAll on a pipeline; in-tree edits use add (TestQPHelper.java:163-167, TestSpanQueryParser.java:102-111, TestSpanQueryParserSimpleSample.java:98-103). No CHANGES or MIGRATE entry mentions the views. With the fix every upstream queryparser suite passes.

The strongest case for "intended" (built by the devil's-advocate verifier; the cross-model reviews rated it weak): read literally, the javadoc promises propagation only "when a QueryConfigHandler object is set on" the pipeline, not on each insertion; the mutators' javadocs are bare @see List#add and promise nothing; QueryNodeProcessor says the handler "should be set using ... setQueryConfigHandler(QueryConfigHandler) before ... process(QueryNode) is invoked" (QueryNodeProcessor.java:32-34), which puts the duty on the caller; java.util.List does not require views to go through overridden methods; editing StandardQueryParser's pipeline needs a cast; stock processors tolerate a missing config on purpose; and the workaround is one call.

Why it fails: here the pipeline has a config and every neighbouring processor carries it, and the processor arrives through the pipeline's own public List API, so the pipeline is the caller that owes it the config, as its five mutators and addProcessor show. The same failure happens on new QueryNodeProcessorPipeline(config) with no cast. The null tolerance is meant for a pipeline without a config; here it turns a missing config into silently wrong queries. Calling the documented setters again does not repair it, only re-setting the handler does, and none of the checked documentation says so. A maintainer could choose to document a limitation (fix option D), but that is not evidence of intent.

Option A was written and validated for this doc with gradle on 645d4fe14a20. B is the smallest change and was run against the repro only; C is the hunt's earlier sketch and its suggested variant; D is listed because a reviewer may ask. Add the new test with whichever is chosen. Users can work around it today by calling setQueryConfigHandler or setQueryNodeProcessor again after editing.

listIterator(int) returns a ConfiguringListIterator around this.processors.listIterator(index): it forwards every method and does in.set(p); configure(p); and in.add(p); configure(p);. subList returns a ConfiguringSubList extends AbstractList<QueryNodeProcessor> implements RandomAccess around this.processors.subList(from, to): get and size forward; set, add(int, p) and remove(int) call the backing sub list, plus configure(p) for set and add and modCount++ for add and remove, so AbstractList builds its iterators, nested sub lists, addAll and replaceAll on them. listIterator() delegates to listIterator(0); replaceAll and the Collections helpers are fixed through it. Because both wrappers delegate to the ArrayList's own views, a stale view still throws ConcurrentModificationException after a direct pipeline.add or remove.

Full diff: +119/-3 in QueryNodeProcessorPipeline.java.

@Override
public ListIterator<QueryNodeProcessor> listIterator(int index) {
  return new ConfiguringListIterator(this.processors.listIterator(index));
}

@Override
public List<QueryNodeProcessor> subList(int fromIndex, int toIndex) {
  return new ConfiguringSubList(this.processors.subList(fromIndex, toIndex));
}

/** Sets this pipeline's handler on a processor that a view put on the pipeline. */
private void configure(QueryNodeProcessor processor) {
  processor.setQueryConfigHandler(this.queryConfig);
}

Trade-offs

• About 120 lines of plain delegation (two private nested classes).
• The view's set hands out the config even when it writes the same processor back (the pipeline's own set skips that case), so List.sort, which rewrites every element through listIterator().set, re-sets each processor's config to the pipeline's. That only matters to code that gave one processor a different config on purpose.
• Like today, insertions through a view do not go through a subclass's override of add or set.
• The sub list stays RandomAccess, as ArrayList's sub list is.

What the checks validated

• Scratch worktree at 645d4fe14a20, gradle, JDK 25: the new test fails 4 of 5 on stock (assertions on and off) and passes 5 of 5 with the fix; the whole :lucene:queryparser:test passes with assertions on (seed 5EED0966) and off (seed 5EED0967): 31 suites, 538 tests, 0 failures, 1 skipped (the 30 upstream suites include TestQPHelper, TestStandardQP, TestMultiFieldQPHelper, TestPointQueryParser, TestPrecedenceQueryParser, TestSpanQueryParser and TestSpanQueryParserSimpleSample). :lucene:queryparser:check -x test (checkGoogleJavaFormat, ecjLint, forbiddenApis, the regenerate checksum checks) and renderJavadoc pass.
• With the fixed class shadowed in the standalone runner: the finder's test 2 of 2, the independent reproducer's 9 of 9 and its fail-fast test 2 of 2 (a stale listIterator() and subList() still throw ConcurrentModificationException); the model-based test: 0 of 63 operations break the config invariant, 0 content mismatches (10,125 judged sequences, 30,375 with its two controls; its mutant control is still caught); the stock differential: 147 of 147 cases equal the oracle.

Three lines in QueryNodeProcessorPipeline.process (:77-79): before running each processor, set the pipeline's handler on it if it differs.

for (QueryNodeProcessor processor : this.processors) {
  if (processor.getQueryConfigHandler() != this.queryConfig) {
    processor.setQueryConfigHandler(this.queryConfig);
  }
  queryTree = processor.process(queryTree);
}

Trade-offs

Covers every insertion path, including future ones, with the smallest change. But it overrides a config that code set on one processor on purpose, writes to processors on every parse, and leaves the state wrong between the edit and the first parse: an inserted processor still reports a null config until then.

What the checks validated

Run against the repro only (shadowed, assertions on): the 3 parse tests pass, testListIteratorAddSetsHandler still fails (4 of 5 pass). The module tests were not run for this option.

The hunt's sketch returns a fresh AbstractList view per call whose set, add and remove call the pipeline's own methods. A variant the verifiers and the audit suggested: let QueryNodeProcessorPipeline extend AbstractList and bump modCount in its own mutators.

Trade-offs

The sketch loses fail-fast: an independent reproducer showed (executed) that a stale listIterator or subList no longer throws after a direct pipeline.add or remove, and the stale sub list silently returns elements. The AbstractList variant would keep fail-fast but changes equals/hashCode from identity (the pipeline overrides neither today) to element equality.

What the checks validated

The sketch passes the finder's and the reproducer's tests, the history probe, the model-based test (0 violations), the stock differential (147 equal) and 150 upstream flexible-parser tests; its fail-fast check fails 2 of 2. The AbstractList variant was not run.

Add a note to the class and view javadocs, and point to the workaround.

Trade-offs

No code change, but it keeps the silent cases and undercuts the List API that LUCENE-1937 added for editing pipelines.

Method: an executed contract probe, then bounded-exhaustive model-based testing (registry: non-formal, because the first signal was the probe; the finder tagged it formal). A hunt unit, flexible processor pipeline insertion paths, 2026-10-28.

• Probe (first signal). A round-13 planner read the class javadoc and the overriding mutators and ran 14 insertion paths on the stock main jars: 8 handed the processor the pipeline's config, 6 (listIterator add/set, subList add/set, replaceAll) left it null; on StandardQueryParser's pipeline 4 paths threw NullPointerException at parse.
• Model-based test. PipelineMbt checks a java.util.ArrayList reference model plus the invariant "every processor on the pipeline carries the pipeline's config" after every step of every sequence of up to 2 of 63 List, view, default-method and Collections operations from 3 start pipelines (10,125 judged sequences; 30,375 with its two controls): 0 content mismatches, invariant violations only on 28 view, default-method and Collections operations. Vacuity controls: routed views give 0 violations, a mutant add(int, p) is caught.
• Stock differential. Each of the 21 StandardQueryParser processors replaced through 7 paths, 22 queries, oracle = the same edit through set(k, p) or add(k, p): 24 silent, 54 loud, 69 equal of 147 cases.
• Then a public-API JUnit on the stock main jars, followed by independent reproduction, cross-model reviews, a devil's advocate, an adjudication and an audit (see the checks).