lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erik Hatcher <e...@ehatchersolutions.com>
Subject Re: Multiple Analyzers
Date Mon, 31 Oct 2005 00:50:18 GMT

On 30 Oct 2005, at 11:03, Daniel.Clark@sybase.com wrote:
> I am implementing a search that allows the user to toggle the  
> Soundex and
> Stem functionality on and off.  When the Soundex or Sten functions  
> are not
> on, the search will use the StandardAnalyzer as default.  I  
> implemented the
> Soundex (SoundexAnalyzer from book Lucene In Action) and Stem
> (SnowballAnalyzer) analyzer successfully individually.  They only  
> work if I
> use the analyzers during both indexing and searching.  The problem  
> I have
> is that I seem to have to use one of the three analyzers or  
> nothing.  I
> heven't been able to combine them.  The following may help better to
> understand what I am trying to achieve.  Any help would be greatly
> appreciated.
>
> Indexing
> ========
> Index using recursive TokenStream object.
>
> public TokenStream tokenStream(String fieldName, Reader reader) {
>       TokenStream result = new SoundexFilter (
>                         new SnowballFilter (
>                            new StopFilter (
>                                                                        
>  new
> LowerCaseFilter (
>
> new StandardFilter (
> new StandardTokenizer(reader))),
>
> StandardAnalyzer.STOP_WORDS)));
>       return result;
> }

So you're using a single analyzer for all fields during indexing?

> Searching
> ==========
> If user selects Soundex function, use SoundexAnalyzer in query.
> Else if user selects Stem option, use SnowballAnalyzer in query.
> Else use StandardAnalyzer.

But different analyzers for searching.... you have to be quite  
careful about this sort of thing.  And in this particular case its  
tricky.  You can't stem and soundex during indexing and then  
selectively turn those features off without having indexed the  
various combinations you want available during search.

One way to address this is to index the same text into multiple  
fields with each using a different analysis process.  Then for  
search, line up the query type (stemming, soundex, standard) with the  
proper field.  (side note, this gets even more fun if you're using  
QueryParser and the user enters in "field:term"!).

Another option is to index into multiple indexes - one for each type  
of analysis.  I use this myself for case sensitive and insensitive  
searches.

     Erik


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message