Monitor HighlightsMatch drops a hit that has the same positions as another hit but different offsets (shingles, multi-word synonyms), because Hit.compareTo ignores offsets (since 8.2.0)

HighlightsMatch.MATCHER reports, for every stored query that matches a document, the hits that Weight#matches finds, as Hit(startPosition, startOffset, endPosition, endOffset) records per field. It keeps them in a TreeSet, and Hit.compareTo compares the start and end positions only, although Hit is a record whose equals and hashCode use all four fields. Analyzers that stack tokens of different lengths at one position (ShingleFilter with unigrams, multi-word synonyms through SynonymGraphFilter and FlattenGraphFilter, CJKBigramFilter with unigrams) produce two different hits with equal positions, because the index stores no position length; when a query matches both, the set keeps one. For wi fi with shingles and the query wi OR "wi fi", Weight#matches reports [0(0)->0(2), 0(0)->0(5)] and HighlightsMatch returns [0(0)->0(2)], getHitCount() 1, with no error. Which hit survives depends on clause order. The same comparator makes getHits(field).contains(...) ignore offsets and lets two matches with different hits be equals while their hash codes differ. Which queries match is not affected. The fix is a tie-break on offsets in compareTo; with it alone, one upstream batch test fails because a batch reports offsets -1 for TextField, so the test must index offsets (or the batch path must change).

• Monitor: Lucene's reverse search module (lucene-monitor). You register queries, then monitor.match(doc, matcher) tells you which of them match a document. The matcher decides what you get back; HighlightsMatch.MATCHER returns a HighlightsMatch per matching query with its hits per field.
• Hit: HighlightsMatch.Hit(startPosition, startOffset, endPosition, endOffset), a public record (HighlightsMatch.java:189). It prints as startPosition(startOffset)->endPosition(endOffset): 0(0)->0(5) is position 0 to 0, characters 0 to 5. The hits come from Weight#matches (HighlightsMatch.java:49-95).
• Stacked tokens: tokens at the same position. ShingleFilter(2, 2) keeps unigrams by default, so wi fi gives wi [0,2), wi fi [0,5) and fi [3,5); wi and wi fi are both at position 0. A multi-word synonym guinea pig => cavy (original kept, flattened for indexing) puts cavy [0,10) on guinea [0,6). The index stores no position length, so both hits of a pair span position 0 to 0; only the offsets tell them apart.

1. addHit puts every hit into a TreeSet per field: hits.computeIfAbsent(field, _ -> new TreeSet<>()).add(new Hit(...)) (HighlightsMatch.java:176-179).
2. Hit.compareTo compares startPosition, then endPosition, and stops (HighlightsMatch.java:199-203). For 0(0)->0(2) and 0(0)->0(5) it returns 0.
3. A TreeSet decides that an element is already present when compareTo returns 0 (its javadoc: "two elements that are deemed equal by this method are, from the standpoint of the set, equal"), so the second hit is dropped, although equals says the two differ.
4. The default query decomposer splits wi OR "wi fi" into two queries; when both match, CandidateMatcher.addMatch merges their matches through HighlightsMatch.merge (CandidateMatcher.java:75-85, HighlightsMatch.java:65-67, :142-151), which copies the hits into a new TreeSet and drops the same hit. Without decomposition, the one Weight#matches call loses it in addHit.
5. getHits(field) returns a read-only view of that TreeSet (:125-129), so contains answers through compareTo and ignores offsets; HighlightsMatch.equals compares the hit maps (:159) and ignores offsets too, while hashCode (:164-169) includes them.

On main a9779ef3c193: shingles, wi OR "wi fi" on wi fi: expected [0(0)->0(2), 0(0)->0(5)], got [0(0)->0(2)], getHitCount() 1. Synonym guinea pig => cavy, guinea OR cavy on guinea pig: expected [0(0)->0(6), 0(0)->0(10)], got [0(0)->0(10)]. PrefixQuery wi* over shingles keeps [0(0)->0(5)]. CJKBigramFilter with unigrams on 東京, PrefixQuery 東: Weight#matches gives 0(0)->0(1) and 0(0)->0(2), HighlightsMatch keeps 0(0)->0(2). The matches for wi and wi (hits 0(0)->0(2) and 0(2)->0(4)) are equals, with hash codes 4564 and 6480. Which hit survives depends on clause order (SHOULD(shingle, unigram) keeps 0(0)->0(2), SHOULD(unigram, shingle) keeps 0(0)->0(5)) and, in decomposed queries, on hash iteration order; the loss happened for every seed tried.

• Who is exposed. Applications that use the monitor with HighlightsMatch.MATCHER, for example to show which words of an incoming document matched a stored query, and whose analyzer stacks tokens of different lengths at one position. Every time a stored query matches two such stacked tokens in one document (a word and its multi-word synonym, a prefix query that matches a unigram and a shingle, a disjunction of both), one hit is silently dropped and getHitCount() is one too low.
• Opt-in on both sides. The caller picks the matcher on every Monitor.match call (Monitor.java:218, :240; the other standard matchers are QueryMatch.SIMPLE_MATCHER, ScoringMatch.DEFAULT_MATCHER and ExplainingMatch.MATCHER), and the analysis chain is the application's. HighlightsMatch.MATCHER is one of the module's documented standard matchers, not an expert API.
• No scale needed. One document, one query, one stacked pair. Single documents, batches whose fields index offsets (a batch of plain TextField documents reports offsets -1, so its stacked hits are identical), and both the plain and the decomposed query paths are affected.
• Not affected. Which queries match a document; the simple, scoring and explaining matchers; hits whose offsets are not recorded (-1), which are truly identical.
• Contract fallout. getHits(field).contains(...) cannot detect wrong offsets (upstream's own offset assertions are affected), and HighlightsMatch objects that are equals can have different hash codes, which breaks hash-based collections.
• How likely. Rare: it needs the matcher, a stacking analyzer and a query that matches two stacked tokens. Nobody has reported it since the TreeSet was introduced in 2015. Priority Low.

# Lucene main 645d4fe14a20, assertions on (-ea), seed DEADBEEF: 1 test(s) run, 1 failed
    => java.lang.AssertionError: [0(0)->0(5)] expected:<2> but was:<1>
       org.junit.Assert.fail(Assert.java:89)
       org.junit.Assert.failNotEquals(Assert.java:835)
       org.junit.Assert.assertEquals(Assert.java:647)
       org.apache.lucene.monitor.TestHighlightsMatchStackedHits.testUnigramAndShingleAtSamePosition(TestHighlightsMatchStackedHits.java:56)

• Contract text. Hit is a public record over (startPosition, startOffset, endPosition, endOffset) (HighlightsMatch.java:181-189). getHits(): "a map of hits per field" (:105-110); getHitCount(): "the total number of hits for the query" (:131-140). The package doc says HighlightsMatch#MATCHER returns "the matching queries along with the matching terms for each query" (package-info.java:46-47); the dropped hit is a different matched term covering different text. No javadoc explicitly promises that every distinct Weight#matches hit is kept: the expectation rests on the record's equals, this wording and the JDK contracts, a strong reading rather than a quoted guarantee.
• JDK contracts. Comparable strongly recommends a natural ordering consistent with equals because sorted sets "behave 'strangely'" otherwise; the TreeSet javadoc says a set with an inconsistent ordering "fails to obey the general contract of the Set interface"; Set.contains and Object.hashCode are specified through equals.
• Tests. TestHighlightingMatcher asserts exact hits with getHits(...).contains(new HighlightsMatch.Hit(...)) 16 times (14 on FIELD, 2 on "field1"/"field2" at lines 108-109; for example line 63, Hit(3, 10, 3, 14)). The tests treat offsets as part of a hit, but these assertions cannot catch a wrong offset today: they fail when no hit has the expected positions, never when only the offsets differ; the batch assertion at line 541 does not even hold (a batch returns offsets -1 for TextField), and nobody could notice. testIdenticalMatches (lines 182-198) is the deliberate dedup: the same term as MUST and SHOULD, hits identical in positions and offsets, getHitCount() 1; it passes with the fix.
• History. In Luwak, the library the module was imported from, compareTo was already position-only while hits lived in lists; commit 95e9eaf5b6 (2015, "Decompose DisjunctionQueries, and ensure highlights are only reported once") switched to TreeSets with only the identical-hit test. Hit.equals was hand-written over all four fields, and flaxsearch/luwak#87 (2016) added a hashCode "compatible with the existing equals()". The donation (LUCENE-8766) and the record conversion (#13328, 10.0.0) kept all of it. No CHANGES, MIGRATE or javadoc text says that hits with equal positions are merged.

The intended-behaviour checker built the strongest case for "dedup by position is the design": the class javadoc says it "contains the hit positions of a matching Query" (HighlightsMatch.java:36); the position-keyed TreeSet was added on purpose; Comparable only recommends consistency with equals; core's DisjunctionMatchesIterator also orders by position (DisjunctionMatchesIterator.java:197-206); and the index has no position length anyway.

Why the evidence weighs against it: the dedup commit targeted hits reported twice by decomposed clauses, which are identical in positions and offsets and stay merged with the fix (its test is no longer decomposed today, because QueryDecomposer keeps a MUST plus SHOULD query whole, QueryDecomposer.java:93-95, but it still checks that identical hits merge); the author made offsets part of a hit's identity (equals, then hashCode); the survivor depends on clause order, which no dedup policy (such as "keep the widest span") would do; and the result breaks the Set and equals/hashCode contracts and blinds the module's own offset assertions. No document promises distinct same-position hits, so this is a judgement on the evidence. The checker rated the argument weak, a cross-model reviewer moderate; the adjudication confirmed the bug with high confidence.

Two variants of the same one-method fix were built and run through gradle; they differ only in how they handle the one upstream test that the fix exposes. The other options were not built.

Hit.compareTo keeps comparing start and end positions, then compares startOffset and endOffset (+6/-1 in HighlightsMatch.java). With this alone, TestHighlightingMatcher.testHighlightBatches fails at line 541, assertTrue(m1.getHits(FIELD).contains(new HighlightsMatch.Hit(1, 4, 2, 16))): a TextField document matched in a batch of two or more documents is indexed with its own field type and gets offsets -1, while a single document goes through a MemoryIndex that records offsets. Today the assertion passes only because contains ignores offsets. The test change gives the test's three batch documents a field type that indexes offsets, so the asserted hits are what the batch really returns (all three: a batch must have one schema).

+      // a batch is indexed with the documents' own field types, so index offsets to get them back
+      FieldType withOffsets = new FieldType(TextField.TYPE_NOT_STORED);
+      withOffsets.setIndexOptions(IndexOptions.DOCS_AND_FREQS_AND_POSITIONS_AND_OFFSETS);
       Document doc1 = new Document();
-      doc1.add(newTextField(FIELD, "the cell biology count", Field.Store.NO)); // matches
+      doc1.add(new Field(FIELD, "the cell biology count", withOffsets)); // matches
       Document doc2 = new Document();
-      doc2.add(newTextField(FIELD, "nope", Field.Store.NO));
+      doc2.add(new Field(FIELD, "nope", withOffsets));
       Document doc3 = new Document();
-      doc3.add(newTextField(FIELD, "biology text", Field.Store.NO));
+      doc3.add(new Field(FIELD, "biology text", withOffsets));

(plus the imports of FieldType, TextField and IndexOptions.)

     public int compareTo(Hit other) {
       if (this.startPosition != other.startPosition)
         return Integer.compare(this.startPosition, other.startPosition);
-      return Integer.compare(this.endPosition, other.endPosition);
+      if (this.endPosition != other.endPosition)
+        return Integer.compare(this.endPosition, other.endPosition);
+      // consistent with equals: the hit TreeSets must not drop a hit that differs in offsets only
+      if (this.startOffset != other.startOffset)
+        return Integer.compare(this.startOffset, other.startOffset);
+      return Integer.compare(this.endOffset, other.endOffset);
     }

Trade-offs

• Hit is public and Comparable: its natural ordering gains a tie-break, but only among hits with equal positions, which cannot coexist in a HighlightsMatch today.
• getHitCount() and getHits(field) report one more hit at each stacked position: the point of the fix, but visible; a CHANGES entry under bug fixes fits.
• Hits without recorded offsets (-1) stay identical and are still merged, so fields without offsets keep today's counts.
• The test change leaves the batch question open: whether a batch should record offsets for TextField is a separate, disputed behaviour.

What the checks validated

• Stock a9779ef3c193: the new test fails 4 of 4 with assertions on and off. With the tie-break: 4 of 4 pass.
• Tie-break alone: TestHighlightingMatcher 20 tests, 1 failure (testHighlightBatches, line 541).
• Tie-break + test change: all of :lucene:monitor:test with assertions on, 33 classes, 185 tests (181 existing, 4 new), 1 skipped, 0 failures; the new test and TestHighlightingMatcher with assertions off, 24 tests, 0 failures; :lucene:monitor:check -x test (google-java-format, ecjLint, forbiddenApis) passes. The test change alone on stock: TestHighlightingMatcher 20 of 20 pass.
• Earlier, in the hunt: the finder's test, the independent reproducer's 5 tests, the devil's advocate's 3 probes and the version-portable check (also on 8.2.0) pass with the same tie-break; testIdenticalMatches still passes, so identical hits are still merged.

Instead of changing the test, make the batch path index positions with offsets, as the single-document path does (a sketch from the separate batch-offsets candidate: +87/-2 in DocumentBatch.java); testHighlightBatches stays as it is.

Trade-offs

Bigger, and it settles a separate question on the way: TextField's field type indexes positions but not offsets, and that batch behaviour is already described in the open apache/solr#2382. Better raised separately, after A1.

What the checks validated

• Tie-break + batch change: all of :lucene:monitor:test with assertions on, 185 tests, 1 skipped, 0 failures.

Not built.

Trade-offs

Fixes the loss and the contracts, but drops the position-sorted iteration order that getHits() users get today.

Not built.

Trade-offs

Still needs a comparator consistent with equals for the Set and equals/hashCode contracts, invents semantics, and still loses a matched term.

Not built.

Trade-offs

Leaves the lost terms, the Set contract break and the equals/hashCode inconsistency in place.

Method: code reading (the registry's "non-formal" tier). A hunt unit compared Monitor.match(Document[]) with Monitor.match(Document). While reading TestHighlightingMatcher.testHighlightBatches, the finder saw that its assertion on offsets (line 541) could never fail, and traced that to Hit.compareTo and the TreeSet.

• The finder wrote a public-API JUnit with Weight#matches on a MemoryIndex as the reference (shingles, wi OR "wi fi"), which fails on main with and without assertions, on branch_10_6 and 10.5.2, and passes with the tie-break.
• It replayed the upstream monitor suite with the tie-break in its own harness: 179 of 180 pass, the one failure being testHighlightBatches (the batch-offsets behaviour); 180 of 180 together with the batch change. (The 185 tests in the fix section come from a separate gradle run of the whole module, 181 existing tests plus the 4 new ones; the two counts are not comparable.)
• Independent checks followed (see "Independent checks"), then a version sweep and this doc's gradle validation.