Word2VecModel: a term listed twice shifts every later term onto another term's vector

Word2VecModel.addTermAndVector stores each entry at the next array slot (termsAndVectors[loadedCount++]) and calls BytesRefHash.add(term) without looking at its result (Word2VecModel.java:58-62 on main a9779ef3c193). vectorValue(BytesRef term) then reads the slot whose index is the term's hash id (:69-74). The two numbers agree only while every term is new: for a term that is already present, BytesRefHash.add allocates no id and returns -(id)-1, while the slot counter still advances. From then on, every later distinct term reads the slot one before its own line (one more slot back for each further repeat). For the model a (1,0), a (1,0), b (0,1), c (-1,0), vectorValue("b") returns (1,0) and vectorValue("c") returns (0,1). Through Word2VecSynonymFilterFactory with its defaults, a Deeplearning4j-format zip listing car, car, auto, banana, fruit loads without any error, and banana is expanded to [auto, car, car] instead of [fruit]. The trigger is a model that lists some term twice: a hand-built or converted model file, or code that calls addTermAndVector with a repeat; stock Deeplearning4j output lists each word once. Every release with the API is affected: 9.7.0 to 9.12.3, 10.0.0 to 10.6.0 and main.

• Word2vec model: a list of words, each with a vector. Word2VecSynonymFilter looks up a token's vector and emits the words whose vectors are nearest, found with an HNSW graph built over the model.
• Slot: the position of an entry in Word2VecModel's array termsAndVectors, one per added line.
• Id: the number BytesRefHash gives a distinct term: 0, 1, 2, ... in the order terms are first added.

1. addTermAndVector (Word2VecModel.java:58-62) normalizes the vector, stores the entry at termsAndVectors[loadedCount++] and calls word2Vec.add(term) without using the returned id.
2. BytesRefHash.add (BytesRefHash.java:310-345) gives a new term the next id; for a term it already holds it allocates nothing and returns -(id)-1 (javadoc :314-316).
3. vectorValue(BytesRef term) (Word2VecModel.java:69-74) reads termsAndVectors[word2Vec.find(term)]: it uses the id as the slot.
4. Word2VecSynonymProvider builds its graph over all size() slots (Word2VecSynonymProvider.java:56-67; size() returns the header's dictionarySize, Word2VecModel.java:86-88), takes the query vector from vectorValue(term) (Word2VecSynonymProvider.java:78) and names each hit by slot with termValue(id) (Word2VecSynonymProvider.java:97). So a later term is searched with the previous term's vector, while its own vector stays in the graph and can still come back as a synonym of other words.
5. Dl4jModelReader.read() (Dl4jModelReader.java:64-81) adds every line of the syn0 file with no repeat check. Its only deliberate check is the vector dimension (:99-109). Other malformed files fail loudly but incidentally, with low-level exceptions: more lines than the header declares, fewer lines, or a plain-text first term shorter than 3 characters (StringIndexOutOfBoundsException from the base64-prefix check, :73-75). So the a, a, b, c model loads from a file only with base64-encoded terms; the file example below uses car.

When the capacity counts every line, as here, every index stays in bounds, so nothing fails. Each further repeat shifts the later terms one more slot.

In the banana row the synonyms are auto's nearest neighbours: cosines 0.20, 0 and 0 with banana, while fruit's is 0.995. The result is the same whether the 0.8 threshold applies to the cosine or to Lucene's (1 + cos) / 2 score. car appears twice because both lines got their own graph node; that comes from storing the repeat at all, not from the shift.

• Who is hit. Users of the @lucene.experimental word2vec synonym support: the Word2VecSynonym filter factory, Word2VecSynonymFilter with a Word2VecSynonymProvider, or Word2VecModel directly. No default analyzer uses it.
• The trigger. A model that lists some term twice, with at least one distinct term after it: a model file built or converted by hand or by a script (the reader accepts any syn0 file in Deeplearning4j's text layout; lower-casing a cased vocabulary while converting, which turns Car and car into two car lines, is one plausible way, not observed in a real file), or code that calls the public addTermAndVector with a repeat.
• Not stock Deeplearning4j output. Its writer emits one line per vocabulary word, and Lucene's own sample model in the tests has 235 lines and 235 distinct words. Repeated words do occur in some word2vec-format files elsewhere: gensim's loader warns about them and keeps the first.
• What goes wrong. Every distinct term after the first repeat is looked up with another term's vector, so wherever the filter is configured (index or query time) it is expanded with that term's synonyms (the visible change can be small when the two vectors happen to be close). With one repeat early in a large vocabulary, that is almost the whole vocabulary. No error or warning.
• Not affected. Models with unique terms. Terms before the first repeat, and all terms when the only repeat is on the last line, keep their own vectors; the repeated line is still a graph node, so the repeated word can come back twice among other words' synonyms (the second car above). No index corruption and no exception; the only extra cost is one graph node per repeated line.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: expected b's vector [0.0, 1.0], actual [1.0, 0.0]
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.assertTrue(Assert.java:42)
       org.apache.lucene.analysis.synonym.word2vec.TestWord2VecModelRepeatedTerm.testRepeatedTermKeepsLaterVectors(TestWord2VecModelRepeatedTerm.java:37)

• The contract. Class javadoc: "Word2VecModel is a class representing the parsed Word2Vec model containing the vectors for each word in dictionary" (Word2VecModel.java:26-28). The upstream test TestWord2VecSynonymProvider.testModel_shouldReturnNormalizedVectors (TestWord2VecSynonymProvider.java:110-121) builds a model through the same public API and asserts that vectorValue(term) is that term's normalized vector. BytesRefHash.add documents the negative return for bytes already present (BytesRefHash.java:314-316).
• No precondition. addTermAndVector and vectorValue(BytesRef) have no javadoc; the constructor, the package doc, the factories, CHANGES.txt and MIGRATE.md say nothing about unique terms.
• The damage falls on other terms. A repeat does not decide which of the two a vectors wins; it silently changes b and c, which are listed once.
• Other malformed files fail loudly. A wrong vector dimension (the reader's one deliberate check) gives "Word2Vec model file corrupted"; more lines than the header declares give ArrayIndexOutOfBoundsException and fewer give NullPointerException when the provider is built (both incidental, low-level failures). Of the cases probed, only a repeat is silent.
• History. The first revision of #12169 (2abb1ea6de1, 2023-02-23) kept entries in a HashMap<BytesRef, TermAndVector> with put/get, so lookups stayed correct per term: a repeat replaced only that word's map entry (the last vector won), although both lines still took an array slot and a graph node. After a review question ("did you consider using BytesRefHash?"), PR commit bf712b90215 (2023-04-05) switched to BytesRefHash and used the id as the slot without using add's return value. No comment on the PR or on issue #12168 discusses repeated terms. On the 9.7.0 jars, putting back only the HashMap lookup makes the hunt's tests pass.

The case for "intended": a word2vec model is a dictionary with one vector per word, and Dl4jModelReader is documented to read "the file generated by the library Deeplearning4j" (Dl4jModelReader.java:34-36), which never repeats a word. The model is trusted configuration and the package is @lucene.experimental, so a file with a repeated word is malformed input: garbage in, garbage out.

Why it fails: no precondition is written anywhere, and Word2VecModel has a public constructor and a public addTermAndVector that upstream tests call directly. @lucene.experimental means the API may change, not that results are undefined. "Garbage in" would allow a wrong vector for the repeated word, not for every unrelated word after it. Other malformed files do fail loudly in the reader, and the PR's first revision looked up every term correctly despite repeats, so the shift is an accident of the BytesRefHash switch, not a design. A reviewer from another model family (OpenAI GPT-5.6) rated the case "moderate" and rejected it; the devil's advocate and the adjudication rejected it too. What survives is a severity point: the documented producer writes unique words, so the reach is low.

Two options were built and validated in a scratch worktree of a9779ef3c193; A is recommended.

Use the id that BytesRefHash.add returns and throw when it is negative, so today's numbering (slot = id) holds. The capacity check runs before add, so a model that is already full leaves the hash unchanged; it throws IndexOutOfBoundsException, the superclass of the ArrayIndexOutOfBoundsException thrown today for too many lines. +11/-1 in Word2VecModel.java, including the java.util.Objects import.

  public void addTermAndVector(TermAndVector modelEntry) {
    modelEntry = modelEntry.normalizeVector();
    // check the capacity before add() changes the hash
    Objects.checkIndex(loadedCount, termsAndVectors.length);
    int termOrd = this.word2Vec.add(modelEntry.term());
    if (termOrd < 0) {
      // vectorValue(BytesRef) uses the hash id as the array slot, so a repeated term would shift
      // every later term onto the previous term's vector
      throw new IllegalArgumentException(
          "Duplicate term in word2vec model: " + modelEntry.term().utf8ToString());
    }
    assert termOrd == loadedCount;
    this.termsAndVectors[loadedCount++] = modelEntry;
  }

Trade-offs

• Loud and early, like the reader's handling of a wrong dimension; the message names the term. Through the factory the exception surfaces when the analyzer is built, and nothing is cached for that model name (Word2VecSynonymProviderFactory.java:43-51 caches only after a successful load).
• Compatibility. A model file with a repeated word that loads today, with silently wrong synonyms, fails to load after the fix; the user has to remove the repeat. Direct callers of addTermAndVector that add a repeat also start getting the exception, and a repeat added once the model is already full throws the IndexOutOfBoundsException, not the duplicate IllegalArgumentException, because the capacity check comes first. Worth a CHANGES entry.

What the checks validated

• stock: the repro fails 2 of 2 with assertions on and off; the word2vec test package (5 classes, 23 tests with the repro) fails only the repro's 2 tests;
• fix A: the repro passes 2 of 2 with assertions on and off; the word2vec package passes (23 tests, 0 failures); the whole lucene/analysis/common test task, repro included, runs 340 classes, 1,911 tests, 0 failures, 18 skipped (13 in hunspell classes that need external dictionaries); ecjLint, checkGoogleJavaFormat, forbiddenApisMain and forbiddenApisTest pass; tidy changes nothing;
• the hunt's 3-test repro with fix A shadowed passes 3 of 3; the independent reproducer's test fails its 5 repeat tests with "Duplicate term in word2vec model: red" (or pear), by design (they expect the first vector to be kept), and its 3 controls pass.

Store the entry at the id add returns, ignore a repeat, and let size() return word2Vec.size() so the HNSW graph and its ordinals cover exactly the distinct terms.

  public void addTermAndVector(TermAndVector modelEntry) {
    modelEntry = modelEntry.normalizeVector();
    int termOrd = this.word2Vec.add(modelEntry.term());
    if (termOrd < 0) {
      return; // repeated term: keep its first vector
    }
    this.termsAndVectors[termOrd] = modelEntry;
    loadedCount++;
  }

  @Override
  public int size() {
    return word2Vec.size();
  }

Trade-offs

• Loads such files the way gensim does (it keeps the first and warns), but silently: a later, different vector for the same word is dropped without notice.
• size() changes from the header's count to the number of distinct terms. It has to: with the header count, the graph builder would read empty slots.
• A truncated file with unique words (fewer lines than the header declares), which fails today with NullPointerException when the provider is built, then loads silently as a smaller model.
• No capacity check before add: one line too many still throws ArrayIndexOutOfBoundsException, but only after add has given the term an id that has no slot.

What the checks validated

• fix B: the repro passes 2 of 2 (gradle, assertions on; classpath runner, assertions off); the word2vec package passes (23 tests, 0 failures, including TestDl4jModelReader's size() == 235 on Lucene's sample model); tidy changes nothing;
• the hunt's 3-test repro, the independent reproducer's 8 tests and the second verifier's 3 tests pass with fix B shadowed. The whole module and the lint tasks were not run for B.

Store a repeat at -(id)-1, overwriting the first entry, with size() as in the keep-first option. The PR's first revision (HashMap.put) also let the last vector win for lookups, but it is not the same: it still wrote every line to the next array slot, kept size() at the header's count and named hits by slot, so both lines stayed graph nodes.

Trade-offs

Same as keep-first.

Reject repeats while reading the model file.

Trade-offs

Leaves direct callers of the public addTermAndVector exposed.

Method: code reading plus a small executed probe (the hunt's non-formal tier; the hunt unit tagged it semi-formal because of the later exhaustive check).

• The hunt's planning read the word2vec classes, named the lead "duplicate model terms shift vectors", and a probe on the stock main jars showed a small model with a repeated term returning the wrong vector for both terms after the repeat.
• The hunt unit compared Word2VecModel, Word2VecSynonymProvider and Word2VecSynonymFilter on the real jar with an exact reference written from the javadoc, over 321,075 small models: 49,635 with distinct names (2-D and 3-D direction grids) and 271,440 with names from {a, b, c}, vectors from 4 directions and lengths 2 to 5. Oracle "vectorValue(term) is a vector that was added for that term": 164,160 of 1,525,824 checks over all 321,075 models fail, every failure in a model where a name repeats before a later distinct name.
• It then wrote a 3-test public-API JUnit that fails on the stock jar. An independent reproducer, a second verifier and an intended-behaviour devil's advocate confirmed it with their own tests, and a reviewer from another model family (OpenAI GPT-5.6, reading only) rated it real.