lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Rowe <sar...@gmail.com>
Subject Re: Tokens produced by Shingle filter are not added in the query
Date Mon, 24 Jul 2017 17:38:00 GMT
Hi hariram,

There may be other problems, but at a minimum you have two different analysis classes here.
 You’re printing the output stream from one (CustomSynynymAnalyzer, the source of which
is not shown in your email), but constructing a query from a different one (CustomAnalyzer).

--
Steve
www.lucidworks.com

> On Jul 24, 2017, at 10:53 AM, hariram ravichandran <hariramravichandar@gmail.com>
wrote:
> 
> I'm using Lucene 4.10.4 and trying to construct (shingles) combinations of
> tokens.
> 
> 
> Code:
> 
> public class CustomAnalyzer extends Analyzer {
>    @Override
>    protected Analyzer.TokenStreamComponents createComponents(final String
> fieldName, final Reader reader) {
>        final WhitespaceTokenizer src = new
> WhitespaceTokenizer(getVersion(), reader);
>        TokenStream tok = new ShingleFilter(src, 2, 3);
>        tok = new ClassicFilter(tok);
>        tok = new LowerCaseFilter(tok);
> //        tok = new
> SynonymFilter(tok,SynonymDictionary.getSynonymMap(),true);
>        return new Analyzer.TokenStreamComponents(src, tok);
>    }
> }
> 
> public class Test {
>    public static void main(String[] args) throws Exception {
>        CustomSynonymAnalyzer analyzer = new CustomSynonymAnalyzer();
>        String queryStr = "cup board";
>        TokenStream ts = new CustomAnalyzer().tokenStream("n", new
> StringReader(queryStr));
>        ts.reset();
>        System.out.println("Tokens are :");
>        while (ts.incrementToken()) {
>            System.out.print(ts.getAttribute(CharTermAttribute.class) + ",
> ");
>        }
>        QueryParser parser = new QueryParser("n", analyzer);
>        Query query = null;
>        query = parser.parse(queryStr);
>        System.out.println("\nQuery is");
>        System.out.print(query.toString());
>    }
> }
> 
> 
> 
>> Output:
>> Tokens are :
>> cup, cup board, board
>> Query is n
>> n:cup n:board
>> 
> 
> Tokens are printed as expected. And expecting the resulting query to be *n:cup
> n:board n:cup board*. But tokens formed by shingle filter are not appended
> to the query. I get only *n:cup n:board.* Where is my mistake?
> 
> Thanks.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message