The stale line sits in a <pre> block of the compile() javadoc. Javadoc never compiles such blocks, so when 9.12 changed the constructors nothing in the build failed, and Lucene's own code already uses the new form.
- An FST is Lucene's compact map from byte strings to outputs; term dictionaries and suggesters use it. You build one with
FSTCompiler: add keys in sorted order, then callcompile(). compile()returns only the FST's metadata (null if the FST accepts nothing). The FST bytes went to the compiler's output: an in-memory one by default, or your own file, such as anIndexOutput, set withBuilder.dataOutput(...).- To use an FST you wrote to a file, you open the file again and wrap it: on heap with
new FST<>(metadata, dataInput), or off heap withFST.fromFSTReader(metadata, new OffHeapFSTStore(indexInput, offset, metadata)).
FST.fromFSTReader with the 3-argument OffHeapFSTStore is the form Lucene's own code uses. Select the still to open it at full size. Generated with AI, checked against the code.- You stream a small FST (
cat -> 5,dog -> 7) to your ownIndexOutputwithFSTCompiler.Builder.dataOutput(out), callcompile(), and open its javadoc to see how to read the FST back. - The javadoc says to open the matching input, "then pass it to the FST construct" (an unfinished sentence, no full stop), and shows
fst = new FST<>(fstMetadata, dataInput, new OffHeapFSTStore());. You expect that to compile. - It does not, on 9.12.0 to 10.6.0 and main: javac reports
constructor OffHeapFSTStore in class OffHeapFSTStore cannot be applied to given types;, followed byrequired: IndexInput,long,FSTMetadata<?>andfound: no arguments. The 3-argumentFSTconstructor is gone as well; javac reports only the inner error. - A 2024 change (#13524, in 9.12.0) removed both constructors and moved all of Lucene's callers of those constructors to
FST.fromFSTReader(md, new OffHeapFSTStore(in, offset, md)), but not this javadoc. With that form and a null check, the example compiles and reads back 5 and 7.
Developers who stream an FST to their own file with the public Builder.dataOutput (custom codecs, suggesters, dictionaries) and start from this javadoc, the only off-heap read-back example in the util/fst docs. They get one compile error on each of the 17 releases from 9.12.0 to 10.6.0 and on main. Opt-in, experimental API; documentation only.
Mechanism, terms and numbers445 words
FSTCompiler.compile() returns only an FST's metadata, and its javadoc explains how to turn that into an FST. For an FST written to an ordinary DataOutput such as an IndexOutput (set through the public FSTCompiler.Builder.dataOutput), it shows fst = new FST<>(fstMetadata, dataInput, new OffHeapFSTStore()); (FSTCompiler.java:993). Since 9.12.0 neither constructor exists: OffHeapFSTStore has only (IndexInput, long, FST.FSTMetadata<?>), and FST has only the public on-heap FST(FSTMetadata, DataInput) and a package-private FST(FSTMetadata, FSTReader).
Copied into code, the example fails with one javac error on every release from 9.12.0 to 10.6.0 and on main; on 9.10.0 and 9.11.1, where it was written, it compiles. #13524 removed both constructors on purpose, moved every caller of those constructors to FST.fromFSTReader(md, new OffHeapFSTStore(in, offset, md)), and left this javadoc as it was. The sentence before the example is also unfinished ("pass it to the FST construct", with no full stop). Documentation only: nothing in Lucene runs differently.
| line | text | still valid? |
|---|---|---|
| 973-974 | "NOTE: this will return null if nothing is accepted by the FST themselves." | yes (compile() returns null at lines 1002-1005) |
| 981-984 | first example: fst = FST.fromFSTReader(fstMetadata, fstCompiler.getFSTReader()); | yes |
| 986-989 | "... create the corresponding DataInput, such as IndexInput then pass it to the FST construct" | unfinished sentence, no full stop |
| 993 | fst = new FST<>(fstMetadata, dataInput, new OffHeapFSTStore()); | no: neither constructor exists |
OffHeapFSTStore(IndexInput in, long offset, FST.FSTMetadata<?> metadata)(OffHeapFSTStore.java:39), the only constructor; it readsmetadata.numBytes(line 42).public FST(FSTMetadata<T> metadata, DataInput in)(FST.java:419, loads the bytes on heap), a package-privateFST(FSTMetadata<T>, FSTReader)(line 424), andpublic static FST.fromFSTReader(FSTMetadata, FSTReader)(line 438), which returns null for null metadata.- No
FSTStoretype: #13524 deleted it.
Only the inner error, the missing no-argument OffHeapFSTStore constructor. The missing 3-argument FST constructor is hidden behind it; an independent verifier's ablation (valid OffHeapFSTStore arguments, the documented FST call kept) fails with cannot infer type arguments for FST<> on main, 10.5.2 and 9.12.3.
FST.fromFSTReader(md, new OffHeapFSTStore(in, offset, md)), where offset is the file pointer of the first FST byte: inline with in.getFilePointer() in Test2BFSTOffHeap and FSTDictionary; through a local store variable, followed by skipBytes(store.size()) past the FST bytes, in FSTTermsReader and NRTSuggester; with its stored indexStartFP in the backward-codecs lucene90 FieldReader.
Any runtime defect; the undeclared placeholder names (fstCompiler, dataInput, fstMetadata, fst), normal for a fragment. The class javadoc (lines 64-66) and the Builder.dataOutput javadoc (lines 305-306) say "use the FST constructor to read it", which is still true for the public on-heap constructor.
Exposure and likelihood107 words
- Who. Readers of the
FSTCompiler.compile()javadoc who copy its second example. No query, document or index file reaches it. - Default vs opt-in. Opt-in and expert:
FSTCompiler,FSTandOffHeapFSTStoreare@lucene.experimental. No configuration or special input is involved: the copied example always fails. - Scale. One compile error on 9.12.0 to 9.12.3, on every 10.x release up to 10.6.0, and on main (javac run on each release's own example against its core jar).
- Effect. A compile error, not wrong behaviour. The reader has to find the current form in Lucene's codec code; the util/fst package example (
package-info.java:52) shows only the in-memory path. - Priority. Low.
One JUnit class with one test. It compiles the javadoc example, copied verbatim, with javax.tools against the jars on the test class path; it adds only the imports and the four names the example leaves undeclared. No FST code runs and the test uses no random data; the compile outcome is deterministic.
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.lucene.util.fst;
import java.net.URI;
import java.util.List;
import java.util.Locale;
import javax.tools.Diagnostic;
import javax.tools.DiagnosticCollector;
import javax.tools.JavaCompiler;
import javax.tools.JavaFileObject;
import javax.tools.SimpleJavaFileObject;
import javax.tools.ToolProvider;
import org.apache.lucene.tests.util.LuceneTestCase;
public class TestFSTCompilerCompileJavadocExample extends LuceneTestCase {
public void testOffHeapExampleCompiles() throws Exception {
String source =
"""
import java.io.IOException;
import org.apache.lucene.store.DataInput;
import org.apache.lucene.util.fst.*;
class Example {
void example(FSTCompiler<Long> fstCompiler, DataInput dataInput) throws IOException {
FST.FSTMetadata<Long> fstMetadata;
FST<Long> fst;
fstMetadata = fstCompiler.compile();
fst = new FST<>(fstMetadata, dataInput, new OffHeapFSTStore());
}
}
""";
JavaCompiler javac = ToolProvider.getSystemJavaCompiler();
DiagnosticCollector<JavaFileObject> diagnostics = new DiagnosticCollector<>();
JavaFileObject file =
new SimpleJavaFileObject(URI.create("string:///Example.java"), JavaFileObject.Kind.SOURCE) {
@Override
public CharSequence getCharContent(boolean ignoreEncodingErrors) {
return source;
}
};
List<String> options =
List.of(
"-classpath", System.getProperty("java.class.path"), "-d", createTempDir().toString());
javac.getTask(null, null, diagnostics, options, null, List.of(file)).call();
List<String> errors =
diagnostics.getDiagnostics().stream()
.filter(d -> d.getKind() == Diagnostic.Kind.ERROR)
.map(d -> d.getMessage(Locale.ROOT))
.toList();
// expected: the example at FSTCompiler.java:992-993 compiles; actual: no OffHeapFSTStore()
assertEquals("javac errors in the FSTCompiler.compile() javadoc example", List.of(), errors);
}
}
# $LUCENE: a built checkout (./gradlew jar); $REPRO: the directory holding org/apache/lucene/util/fst/Test...java
# $DEPS: junit 4.13.2, randomizedtesting-runner 2.9.2, hamcrest 3.0
V=11.0.0-SNAPSHOT
CP=$LUCENE/lucene/core/build/libs/lucene-core-$V.jar:$LUCENE/lucene/codecs/build/libs/lucene-codecs-$V.jar:$LUCENE/lucene/test-framework/build/libs/lucene-test-framework-$V.jar:$DEPS
javac -d out -cp "$CP" $REPRO/org/apache/lucene/util/fst/TestFSTCompilerCompileJavadocExample.java
java -ea -Dtests.seed=DEADBEEF -cp "out:$CP" org.junit.runner.JUnitCore org.apache.lucene.util.fst.TestFSTCompilerCompileJavadocExample
# assertions off, same result: java -da -Dtests.asserts=false ...
- Expected
- The copied example compiles.
- Observed
- One javac error, with assertions on and off, on main, 10.6.0 and 9.12.0; the test reports it through
javax.toolsasconstructor OffHeapFSTStore in class org.apache.lucene.util.fst.OffHeapFSTStore cannot be applied to given types;(full text below).
More files, environment, full output224 words
The embedded block was checked mechanically against FSTCompiler.java:992-993 at 645d4fe14a20 (<> unescaped): it is verbatim.
The same test fails on the 10.6.0 and 9.12.0 release jars and passes on 9.11.1, where the example was valid.
No separate regression test is suggested: the test holds a copy of the example, so it would not notice a later edit of the javadoc, and Lucene has no harness that compiles javadoc examples (LUCENE-8488, #9534).
| lucene | main 645d4fe14a20 (11.0.0-SNAPSHOT); the 10.6.0 and 9.12.0 release jars give the same error |
|---|---|
| jdk | OpenJDK 25.0.3 (build 25.0.3+9-LTS) |
| os | Linux aarch64 (a compile-only check; the API mismatch does not depend on the platform) |
# the command above, main 645d4fe14a20, seed DEADBEEF, -ea (-da identical); lines in order, the JDK version line, JUnit banner, vector-module WARNING, NOTE lines and Time: line left out, ... = remaining runner frames
There was 1 failure:
1) testOffHeapExampleCompiles(org.apache.lucene.util.fst.TestFSTCompilerCompileJavadocExample)
java.lang.AssertionError: javac errors in the FSTCompiler.compile() javadoc example expected:<[]> but was:<[constructor OffHeapFSTStore in class org.apache.lucene.util.fst.OffHeapFSTStore cannot be applied to given types;
required: org.apache.lucene.store.IndexInput,long,org.apache.lucene.util.fst.FST.FSTMetadata<?>
found: no arguments
reason: actual and formal argument lists differ in length]>
at __randomizedtesting.SeedInfo.seed([DEADBEEF:742292286A5F4F6]:0)
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:120)
at org.apache.lucene.util.fst.TestFSTCompilerCompileJavadocExample.testOffHeapExampleCompiles(TestFSTCompilerCompileJavadocExample.java:67)
...
FAILURES!!!
Tests run: 1, Failures: 1
# the same test, standalone: 10.6.0 jars -da and 9.12.0 jars -ea: tests_run=1 failures=1, the same single diagnostic; 9.11.1 jars -ea: tests_run=1 failures=0- The javadoc presents it as the way to do it ("To create the FST, you need to:"), and its first example, two lines above, compiles.
- It compiled verbatim on 9.10.0 and 9.11.1; #13524 removed both constructors on purpose, updated every code caller, and missed this javadoc.
- Lucene fixed a non-compiling FST package example as a defect before (LUCENE-6112, 2014).
The full argument170 words
- Offered as working code. "To create the FST, you need to:" followed by two cases; nothing marks the block as pseudo-code (
FSTCompiler.java:976-994). - Correct when written. #12831 (Dzung Bui, 2024-02-06, first released in 9.10.0) added both examples when
FSTstill had the(FSTMetadata, DataInput, FSTStore)constructor andOffHeapFSTStorea no-argument one. The verbatim example compiles on 9.10.0 and 9.11.1. - A missed update, not a choice. On #13524 Adrien Grand (jpountz) called
OffHeapFSTStore"an internal class so it's ok to break the API" (their view). The PR moved every caller of the removed constructors tofromFSTReader, did not touchFSTCompiler.java, and no review comment mentions this javadoc; there is no CHANGES or MIGRATE entry for the removal. - Precedent. LUCENE-6112 (#7174, 2014, fixed) was a compile error in the util/fst package example, reported on java-user@. On #12758 a contributor wrote "The Javadoc could be outdated. Will update it" about
FST.java(before this example existed). Both were surfaced by the finder and a reviewer from another model family (OpenAI GPT-5.6).
The devil's-advocate check133 words
Strongest case for "intended" or "noise": the three types are @lucene.experimental; maintainers called them internal (Adrien Grand on #13524, as quoted; Michael McCandless on #12831 called FSTCompiler "effectively an internal Lucene datastructure", their view); the block already leaves four names undeclared, so it reads like pseudo-code; Lucene does not compile javadoc snippets; the error is loud.
Why it fails: experimental status explains why the API could change, not why the example was left stale. The failure is two constructors that no longer exist, not missing declarations: with the same placeholders the example compiled on 9.10.0 and 9.11.1, and the corrected form compiles today. The package is exported and the classes are not @lucene.internal, and the public Builder.dataOutput javadoc sends users down this path. Every verifier and the adjudication called it real.
Recommended Show the current off-heap form, with a null check (validated)
Replace the stale line with if (fstMetadata != null) { OffHeapFSTStore store = new OffHeapFSTStore(indexInput, offset, fstMetadata); fst = FST.fromFSTReader(fstMetadata, store); } and finish the cut-off sentence (javadoc only). Validated: the patched example compiles on main, 10.6.0 and 9.12.0, reads an FST back end to end, and the util/fst tests pass.
* <pre><code class="language-java">
* fstMetadata = fstCompiler.compile();
- * fst = new FST<>(fstMetadata, dataInput, new OffHeapFSTStore());
+ * if (fstMetadata != null) {
+ * OffHeapFSTStore store = new OffHeapFSTStore(indexInput, offset, fstMetadata);
+ * fst = FST.fromFSTReader(fstMetadata, store);
+ * }
* </code></pre>All options and trade-offs3 options · 543 words
One fix was validated, with the javadoc diff applied to 645d4fe14a20; the alternatives were not run.
Javadoc only, lucene/core/src/java/org/apache/lucene/util/fst/FSTCompiler.java:986-994 (7 lines added, 4 removed in all). The snippet shows the code block; the sentence above it becomes "... you need to first open the corresponding {@link org.apache.lucene.store.IndexInput}, then pass it to an {@link OffHeapFSTStore}, together with the file pointer where the FST bytes start", which also removes the cut-off word "construct". The null check matters: compile() returns null for an FST that accepts nothing, and the OffHeapFSTStore constructor reads metadata.numBytes before fromFSTReader could check for null (a point raised by the cross-model refute-lens reviewer and refined by an independent verifier).
* <pre><code class="language-java">
* fstMetadata = fstCompiler.compile();
- * fst = new FST<>(fstMetadata, dataInput, new OffHeapFSTStore());
+ * if (fstMetadata != null) {
+ * OffHeapFSTStore store = new OffHeapFSTStore(indexInput, offset, fstMetadata);
+ * fst = FST.fromFSTReader(fstMetadata, store);
+ * }
* </code></pre>Trade-offs
None for users: documentation only, no CHANGES entry strictly needed. It keeps the off-heap read the example was written for and matches Lucene's own callers. Caveats worth a clause: the off-heap form needs an IndexInput (an arbitrary DataInput cannot back an OffHeapFSTStore); the store keeps that IndexInput and slices it when the FST is read, so it must stay open while the FST is used; offset must be the first FST byte, and a caller with more data after the FST skips past it itself (as FSTTermsReader and NRTSuggester do); the IndexOutput is normally closed before the IndexInput is opened.
What the checks validated
- The second
compile()example extracted from the patched file and compiled in the same one-test check (declaringindexInputandoffsetinstead ofdataInput): passes 1 of 1 on main with assertions on and off, on the 10.6.0 jars and on the 9.12.0 jars. The stock block fails the same check, so the check detects the bug. - An end-to-end control: an FST with
cat -> 5,dog -> 7streamed to anIndexOutputthroughBuilder.dataOutputand read back exactly as the patched example shows returns 5 and 7 (and null forcow);compile()with no input returns null. 2 tests, 0 failures. ./gradlew :lucene:core:test --tests "org.apache.lucene.util.fst.*" -Ptests.seed=DEADBEEF: 37 tests, 0 failures, 5 skipped (2@Nightly, 3@Ignored)../gradlew :lucene:core:checkGoogleJavaFormatand:lucene:core:renderJavadocpass; the rendered page shows the new code, linksOffHeapFSTStore, and no longer containsnew OffHeapFSTStore()or "FST construct".
fst = new FST<>(fstMetadata, dataInput); (FST.java:419), plus "construct" -> "constructor". Smallest diff; matches the "use the FST constructor" wording of the class and Builder.dataOutput javadocs, and works for any DataInput positioned at the first FST byte (with non-null metadata). It compiles on main and 10.6.0 (the fact check's standalone javac); not run end to end.
Trade-offs
Loads the whole FST onto the heap, so the javadoc no longer shows the off-heap path; it needs the same null check (FST.java:420 reads metadata.numBytes).
The generic wish of LUCENE-8488 (#9534, open since 2018), for example with the javadoc {@snippet} tag backed by compiled code.
Trade-offs
Would stop this class of drift, but it is a build-infrastructure change, not part of this fix.
No formal method. A hunt unit's capped, heuristic sweep compiled the Java code blocks of Lucene's non-sandbox javadocs against the main jars (wrapping each block and adding guessed imports); this block failed with an API-level error. A targeted test then compiled it verbatim next to two controls, and independent verifiers reproduced it with their own tests.
The method in full114 words
- Sweep. The capped inventory sweep of a hunt unit on javadoc examples extracted the
<pre><code class="language-java">blocks of the non-sandboxlucene/**/src/javafiles, wrapped each one heuristically (as a class, members or statements, with guessed imports) and compiled it against the stock main jars withjavax.tools, counting only API-level errors; it flagged the secondcompile()block (2026-10-28). - Targeted test. A public-API JUnit compiled the block verbatim with two controls: the javadoc's own in-memory example and a corrected off-heap form, both of which compile.
- Second error. The missing 3-argument
FSTconstructor, hidden by javac, was first read from the source history and later executed by an independent verifier's ablation. - Tier. Non-formal: it compiles documentation code.
No prior report: GitHub issues and PRs (with the migrated JIRA archive), commits, the dev@, java-user@, issues@ and commits@ lists, downstream repositories and the web were searched on 2026-10-29; re-checked 2026-11-01. No issue or PR names this javadoc, and no open or recent PR touches util/fst.
Searches3 searches
Related, not reports: #12831 wrote the example (valid then); #13524 made it stale and no review comment mentions it; #12758's "The Javadoc could be outdated" is about FST.java and predates the example; LUCENE-6112 (#7174, 2014, fixed) is the same kind of defect in the util/fst package example; LUCENE-8488 (#9534) is a generic wish to compile examples; #15573 is a markup sweep that kept the stale line.
Others file javadoc fixes almost daily (hidingmyname's series, #16904, #16916, #16949); none covers util/fst.
| Where | When | How | Found |
|---|---|---|---|
| Scripted searches, 3 runs | 2026-10-29 | class and method names, the javac error text, the defect class, over GitHub issues, PRs and commits, downstream repositories, dev@, java-user@, issues@, commits@ and the JIRA archive | only #12831 (the origin), FST implementation commits and generic context |
| GitHub, manual | 2026-10-29 | OffHeapFSTStore, "FSTCompiler javadoc", "FST construct", fromFSTReader, fstCompiler.compile(); the files of 189 of 414 open PRs and of PRs closed since 2026-10-20; the review threads of #13524, #12831 and #12758; 7 web searches | no report; no PR touches FSTCompiler.java |
| Re-check | 2026-11-01 | the 63 issues and PRs updated since 2026-10-29 (title and body) and the changed files of their 52 PRs; the stale line on main, branch_11x and branch_10_6 | 0 mentions; 0 PRs touch util/fst; the line is unchanged on all three branches |
Valid when written in 9.10.0 (#12831, 2024-02); stale since #13524 (2024-06-27, first released in 9.12.0) removed both constructors and left this javadoc untouched. Every release since, up to 10.6.0, ships the stale line, and upstream main still has it.
Versions tested6 rows
- 2024-02-06
63d4ba938fe(#12831, Dzung Bui; branch_9xf3e2929a52c):compile()returnsFSTMetadata; the javadoc gets both examples, both correct. First release 9.10.0. - 2024-06-27
126834c09e6(#13524, Armin Braun, approved and merged by Adrien Grand per the GitHub API; branch_9x82b3f8efd49): deletesFSTStore, removes the 3-argumentFSTconstructor, givesOffHeapFSTStoreonly(IndexInput, long, FSTMetadata), changes 10 files, none of themFSTCompiler.java. First release 9.12.0 (and 10.0.0 on the 10.x line). - 2026-01-15
378ca8d1bbf(#15573): changes only the block's markup; the stale line stays. git log -S 'new OffHeapFSTStore()'finds no edit of the line after #12831.
| Version or commit | Verdict | Result |
|---|---|---|
main 645d4fe14a20; upstream main f815e679bf6c | affected | fails with the single error, -ea and -da (upstream main read, not run: the line is unchanged) |
| 10.6.0 | affected | fails with the single error (release jars, -da; annotated tag dated 2026-10-29) |
| 10.0.0 to 10.5.2 | affected | fails, 1 javac error on each of the 12 releases (diagram probe's release scan); the finder's test also ran on 10.0.0 and 10.5.2 |
| 9.12.0 to 9.12.3 | affected | fails, 1 javac error on each (release scan); the finder's test ran on 9.12.0 and 9.12.3 |
| 9.10.0 to 9.11.1 | example valid | the verbatim example compiles (release scan on 9.10.0, 9.11.0 and 9.11.1; the minimal repro passes on 9.11.1) |
| 9.9.2 and older | no such example | not affected: compile() still returned FST<T> (source read; the release scan finds no example in 9.9.0-9.9.2) |
A finder, an independent reproducer, a reviewer from another model family (OpenAI GPT-5.6) with lenses, a devil's-advocate check, an adjudication and an audit. Every executing check reproduced it; the adjudication confirmed it with high confidence as a low-priority documentation defect.
Checks and verdicts215 words
The cross-model reviews could not run code; separate runs re-ran the finder's repro on 10.5.2 with assertions off (fails as expected).
| Check | Verdict | What it found |
|---|---|---|
| Finder | reproduced (doc) | Capped, heuristic compile sweep plus a 3-test class: the off-heap example fails on main (-ea, -da), 10.5.2 and 9.12.3; both controls pass. |
| Refute lens (OpenAI GPT-5.6, reading only) | not a valid review (procedural) | Could not run anything; its reading found 6 points for real and 2 weak counterpoints (experimental API; the fix needs a null check, adopted). |
| Reproducer | real | Own test with an ablation and 2 controls: fails on main (-ea, -da), 10.5.2, 9.12.3 and 9.12.0; compiles on 9.11.1. |
| Intended lens (OpenAI GPT-5.6, reading only) | real, medium | Intended argument weak; found LUCENE-6112 and the #12758 remark. |
| Devil's advocate | real, high | Re-ran on main (-ea, -da); rejected the experimental, internal and pseudo-code arguments. |
| Adjudication and audit | confirmed (high), low | Re-ran the independent reproduction's test with a new seed on main (-ea, -da) and on 9.11.1 (passes); the audit re-ran the finder's test on main. |
| Brief | verified; fix validated | Minimal test fails on main (-ea, -da), 10.6.0 and 9.12.0 and passes on 9.11.1; the patched block passes; the end-to-end control and the util/fst tests pass. |
The diagram as a still image, generated with AI and checked against the code. It is under 10 MB, the limit for a GitHub attachment.
- PNG 420 KB
Finding record, Lucene and upstream sources, repro files and diagram notes1,426 words
- Registry title
- FSTCompiler.compile()'s javadoc shows how to read back an FST written to a non-FSTReader DataOutput with 'fst = new FST<>(fstMetadata, dataInput, new OffHeapFSTStore())', but #13524 removed both the no-argument OffHeapFSTStore constructor and the 3-argument FST constructor in 9.12.0, so the example names APIs that no longer exist (documentation defect)
- Area
- core / util/fst (FSTCompiler.compile() javadoc example)
- Priority
- Low, rank 1259: One stale FSTCompiler.compile() javadoc example fails loudly at compile time and only affects users of the experimental util.fst off-heap API.
- Impact class
- minor
- Impact note
- Documentation only, with no runtime effect: a developer who streams an FST to their own DataOutput (for example an IndexOutput, through the public FSTCompiler.Builder.dataOutput) and copies the compile() javadoc example to read it back off-heap gets a compile error on every release since 9.12.0 and has to find the current form, FST.fromFSTReader(metadata, new OffHeapFSTStore(in, offset, metadata)), in Lucene's own codec code; the util.fst classes are @lucene.experimental, and no configuration, scale or special input is involved.
- Summary
- The javadoc of FSTCompiler.compile() explains how to turn its result into an FST, and for an FST written to an ordinary IndexOutput it shows 'fst = new FST<>(fstMetadata, dataInput, new OffHeapFSTStore());'. Copied into code on any release since 9.12.0 this does not compile: OffHeapFSTStore now needs (IndexInput, offset, metadata) and the 3-argument FST constructor no longer exists, because #13524 changed both and left this javadoc as it was. The working form is FST.fromFSTReader(fstMetadata, new OffHeapFSTStore(indexInput, offset, fstMetadata)), with a null check because compile() returns null for an FST that accepts nothing; Lucene's own codecs and tests already use it, so only the javadoc is wrong.
- Introduced
- 126834c09e6 (#13524 'Reduce overhead for FSTs in FieldReader', by Armin Braun (original-brownbear), approved and merged by jpountz, author date 2024-06-27; branch_9x backport 82b3f8efd49, same author date): it deleted FSTStore and the FST(FSTMetadata, DataInput, FSTStore) constructor, replaced OffHeapFSTStore's implicit no-argument constructor with (IndexInput, long, FSTMetadata), moved every code caller to FST.fromFSTReader(md, new OffHeapFSTStore(in, offset, md)) (Test2BFSTOffHeap, FSTTermsReader, FSTDictionary, NRTSuggester, both FieldReaders), and did not touch FSTCompiler.java (no CHANGES or MIGRATE entry for the removal). The example itself comes from 63d4ba938fe (#12831 'Make FSTCompiler.compile() to only return the FSTMetadata', by Dzung Bui (dungba88), author date 2024-02-06; branch_9x f3e2929a52c; first released in 9.10.0), valid when written; the only later edit to the block is the markup change 378ca8d1bbf (#15573, 2026-01-15) [read: git log -S, git show and git tag --contains in the full-history clone and a Lucene clone; exec: release sweep, see affected_versions] (2024-06-27 (126834c09e6, #13524, author date; PR merged 2024-06-27T07:52Z; the branch_9x backport 82b3f8efd49 has the same author date))
- First affected version
- 9.12.0 [exec] (the first release after #13524); 10.0.0 [exec] is the first 10.x release and affected from the start
- Affected versions
- 9.12.0-9.12.3, every 10.x release from 10.0.0 to 10.5.2, 10.6.0 (released 2026-10-29; branch_10_6 4ae946e5a9b) [exec 2026-11-01: the minimal test fails on the 10.6.0 release jars, -da] and main a9779ef3c193; not affected: 9.10.0-9.11.1 (the verbatim example compiles) and every release up to 9.9.2 (no such example: compile() still returned FST<T>). Not fixed upstream: main 645d4fe14a20, branch_11x, branch_10_6 and branch_10x still carry the stale line (2026-10-29). [exec: the finder's JUnit fails on main (-ea and -da), 10.5.2 (-ea and -da), 10.0.0, 9.12.3 and 9.12.0, while its off-heap test passes on 9.11.1 and 9.10.0 (follow-up history runs); the independent reproduction's JUnit fails on main (-ea and -da), 10.5.2 (-ea and -da), 9.12.3 and 9.12.0 and passes on 9.11.1; the adjudication's re-run fails on main (-ea and -da) and passes on 9.11.1] [read: the example text is identical at 9.10.0, 9.11.1, 9.12.0, 9.12.3, 10.0.0, 10.5.2 and main; the other tags by source read and git tag --contains of 126834c09e6 and 82b3f8efd49]. Caveat: all runs used JDK 25 on Linux aarch64.
- Status
- documented (brief, checks and a per-finding doc exist)
- Verification
- Confirmed by AI-run checks: an independent repro, a cross-model review that could only read the code (its low-confidence “not real” is procedural), an intended-behaviour check, a prior-report search and a skeptical audit; not yet triaged by a person.
- Found by
- non-formal
- Upstream and prior reports
- none found (searched 2026-10-29 and rechecked the same day): three scripted search runs (the finder's 51 queries and two follow-up runs over GitHub issues, PRs and commits, downstream repositories, the dev@, java-user@, issues@ and commits@ lists, the JIRA archive and the web; no high hit reports this javadoc), targeted GitHub searches (OffHeapFSTStore, 'FSTCompiler javadoc', 'FST construct', fromFSTReader, 'fstCompiler.compile()', doc-sweep authors), the files of open PRs (414 listed, the files of 189 checked) and of PRs closed since 2026-10-20, the review threads of #13524, #12831 and #12758, 7 web searches, and lists.apache.org. Read but not the same bug: #12831 (it wrote the example, valid then), #13524 (the change that made it stale; no review comment mentions the compile() javadoc), #12758 (an earlier constructor refactor; its 'Javadoc could be outdated' remark is about FST.java and predates the example), LUCENE-6112 (#7174, 2014, fixed: a non-compiling example in the util/fst package overview, a different site), LUCENE-8488 (#9534, open: a generic proposal to compile-check javadoc examples, naming none), #15225, #13054, #13005, #16272, #15573 (a markup sweep that edited this block and kept the stale line), and the parallel outside javadoc series (hidingmyname #16821-#16828, #16920-#16922, #16927, #16928, #16933-#16936; Deepcodr #16904; Lyin-OTTF #16875, #16916), none of which covers util/fst. Recheck: the 60 issues and PRs updated since 2026-10-27 name neither the class nor the API, and none of their 47 PRs changes FSTCompiler.java; main 645d4fe14a20, branch_11x, branch_10_6 and branch_10x still carry the stale line; the file's latest upstream commit by author date is 8cd74fed97b (#15133, 2026-09-29), which does not touch the block. Tracking issue #16937 does not name it.
- Lucene commit tested
- main
645d4fe14a20(2026-10-28): the one-test repro fails with assertions on and off (standalone, with a plainjavac/javacommand); the 10.6.0 and 9.12.0 release jars give the same single error, and the 9.11.1 jars pass. The fix was checked with the javadoc diff applied to645d4fe14a20: the patched example compiles on main, 10.6.0 and 9.12.0, an end-to-end control reads the FST back, and the util/fst tests,checkGoogleJavaFormatandrenderJavadocpass (2026-11-01, OpenJDK 25.0.3, Linux aarch64).
- Finding record
lucene/core/src/java/org/apache/lucene/util/fst/FSTCompiler.java:972-996 (compile() javadoc: :973 'NOTE: this will return null if nothing is accepted', :983 the FSTReader example, :989 'pass it to the FST construct', :993 'fst = new FST<>(fstMetadata, dataInput, new OffHeapFSTStore());', :996 compile()), :302-316 (Builder.dataOutput javadoc: 'construct the corresponding DataInput and use the FST constructor to read it') and :68 (@lucene.experimental); FST.java:419 (public FST(FSTMetadata, DataInput)), :424 (package-private FST(FSTMetadata, FSTReader)), :438 (fromFSTReader), :62 (@lucene.experimental); OffHeapFSTStore.java:39 (the only constructor, (IndexInput, long, FST.FSTMetadata<?>)) and :28 (@lucene.experimental); Lucene main ea57360912ed FSTCompiler.java is byte-identical (cmp)
- The finding in 30 seconds
a Lucene main checkout (645d4fe14a20) lucene/core/src/java/org/apache/lucene/util/fst/FSTCompiler.java:57-68 (class javadoc, three usage modes, @lucene.experimental), :301-315 (Builder.dataOutput javadoc), :972-995 (compile() javadoc: :973-974, :981-984, :986-989, :991-994, :993), :996-1016 (compile(); null return :1002-1005)a Lucene main checkout (645d4fe14a20) lucene/core/src/java/org/apache/lucene/util/fst/FST.java:62 (@lucene.experimental), :415-421 (public FST(FSTMetadata, DataInput); metadata.numBytes at :420), :423-429 (package-private FST(FSTMetadata, FSTReader)), :431-444 (fromFSTReader; null check :440-442)a Lucene main checkout (645d4fe14a20) lucene/core/src/java/org/apache/lucene/util/fst/OffHeapFSTStore.java:24-30 (@lucene.experimental, implements FSTReader), :39-43 (the only constructor; metadata.numBytes at :42)a Lucene main checkout (645d4fe14a20) lucene/core/src/test/org/apache/lucene/util/fst/Test2BFSTOffHeap.java:94-99, :186-190, :274-278; lucene/codecs/src/java/org/apache/lucene/codecs/memory/FSTTermsReader.java:199-201; lucene/codecs/src/java/org/apache/lucene/codecs/uniformsplit/FSTDictionary.java:97-100; lucene/suggest/src/java/org/apache/lucene/search/suggest/document/NRTSuggester.java:314-316; lucene/backward-codecs/src/java/org/apache/lucene/backward_codecs/lucene90/blocktree/FieldReader.java:92
- Who is affected
a Lucene main checkout (645d4fe14a20) lucene/core/src/java/org/apache/lucene/util/fst/FSTCompiler.java:68, a Lucene main checkout (645d4fe14a20) lucene/core/src/java/org/apache/lucene/util/fst/FST.java:62, a Lucene main checkout (645d4fe14a20) lucene/core/src/java/org/apache/lucene/util/fst/OffHeapFSTStore.java:28 (@lucene.experimental)a Lucene main checkout (645d4fe14a20) lucene/core/src/java/org/apache/lucene/util/fst/package-info.java:52 (only fromFSTReader with getFSTReader())
- Why it looks unintended
a Lucene main checkout (645d4fe14a20) lucene/core/src/java/org/apache/lucene/util/fst/FSTCompiler.java:972-995a Lucene main checkout (645d4fe14a20) lucene/MIGRATE.md and lucene/CHANGES.txt (no entry for the FSTStore / OffHeapFSTStore removal of #13524)
- Suggested fix
a Lucene main checkout (645d4fe14a20) lucene/core/src/java/org/apache/lucene/util/fst/FSTCompiler.java:1002-1005 (compile() returns null); a Lucene main checkout (645d4fe14a20) lucene/core/src/java/org/apache/lucene/util/fst/OffHeapFSTStore.java:42 (metadata.numBytes); a Lucene main checkout (645d4fe14a20) lucene/core/src/java/org/apache/lucene/util/fst/FST.java:419-420a Lucene main checkout (645d4fe14a20) lucene/core/src/java/org/apache/lucene/util/fst/OffHeapFSTStore.java:35-43 (keeps the IndexInput), :54-61 (getReverseBytesReader slices it); lucene/codecs/src/java/org/apache/lucene/codecs/memory/FSTTermsReader.java:201 and lucene/suggest/src/java/org/apache/lucene/search/suggest/document/NRTSuggester.java:316 (skip past the FST); lucene/core/src/test/org/apache/lucene/util/fst/Test2BFSTOffHeap.java:94-95 (output closed before the input is opened)
- History
a Lucene clone: git tag --contains 82b3f8efd49 (first 9.12.0), f3e2929a52c (first 9.10.0)
- TestFSTCompilerCompileJavadocExample.java (one test, public API):
repro/LFV-988/src/org/apache/lucene/util/fst/TestFSTCompilerCompileJavadocExample.java(69 lines)
Made in parallel with this doc from the mechanism in the brief; the PNG is the summary frame. The diagram runs its own small example with three keys (cat -> 5, dog -> 7, dogs -> 12), one more than the steps above.
assets/LFV-988/bug.png(420 KB)