lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Rowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-8137) GraphTokenStreamFiniteStrings does not handle position inc > 1 in multi-word synoyms
Date Thu, 22 Feb 2018 16:47:00 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-8137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16373058#comment-16373058
] 

Steve Rowe commented on LUCENE-8137:
------------------------------------

A test showing the problem - MockSynonymFilter has synonym "cavy" for "guinea pig", and the
anonymous analyzer below has "pig" on its stoplist. QueryBuilder produces a query for only
"cavy", even though the token stream also contains "guinea":

{code:java|title=TestQueryBuilder.java}
  public void testGraphStop() {
    Query syn1 = new TermQuery(new Term("field", "guinea"));
    Query syn2 = new TermQuery(new Term("field", "cavy"));

    BooleanQuery synQuery = new BooleanQuery.Builder()
        .add(syn1, BooleanClause.Occur.SHOULD)
        .add(syn2, BooleanClause.Occur.SHOULD)
        .build();
    BooleanQuery expectedGraphQuery = new BooleanQuery.Builder()
        .add(synQuery, BooleanClause.Occur.SHOULD)
        .build();
    QueryBuilder queryBuilder = new QueryBuilder(new Analyzer() {
      @Override
      protected TokenStreamComponents createComponents(String fieldName) {
        MockTokenizer tokenizer = new MockTokenizer();
        TokenStream stream = new MockSynonymFilter(tokenizer);
        stream = new StopFilter(stream, CharArraySet.copy(Collections.singleton("pig")));
        return new TokenStreamComponents(tokenizer, stream);
      }
    });
    queryBuilder.setAutoGenerateMultiTermSynonymsPhraseQuery(true);
    assertEquals(expectedGraphQuery, queryBuilder.createBooleanQuery("field", "guinea pig",
BooleanClause.Occur.SHOULD));
  }
}
{code}

> GraphTokenStreamFiniteStrings does not handle position inc > 1 in multi-word synoyms
> ------------------------------------------------------------------------------------
>
>                 Key: LUCENE-8137
>                 URL: https://issues.apache.org/jira/browse/LUCENE-8137
>             Project: Lucene - Core
>          Issue Type: Bug
>    Affects Versions: master (8.0), 7.2.1
>            Reporter: Jim Ferenczi
>            Assignee: Jim Ferenczi
>            Priority: Major
>
> The automaton built for graph queries that contain multiple multi-word synonyms does
not handle gaps if they appear in the middle of a multi-word synonym. In such case the token
next to the gap is considered as part of the multi-word synonym. 
> Stop words that appear before or after multi-word synonyms are handled correctly in the
current version but the synonym rule "part of speech, pos" for instance does not create the
expected query if "of" is removed by a filter that is set after the synonym_graph.  One
solution would be to reuse TokenStreamToAutomaton (with minor changes to add the ability to
create token transitions rather than chars) which preserves gaps (as a transition) in the
produced automaton.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message