lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Preserve Original Option In Stemming (EnglishMinimalStemFilterFactory).
Date Tue, 25 Aug 2015 16:45:05 GMT
It's actually a real pain to do this right considering all the different
analysis chains. As Modassar says, the KeywordRepeatFilterFactory
is often "good enough". It'll boost the exact match, but it won't actually
guarantee that only exact-match docs are returned.

Ideally, you'd want the option to turn unstemmed match on and off. But
to do that, you have to have some way to signal the analysis chain when
to emit only the original token at _query_ time. So say there's some rule
like "when you see a $ appended to the term, it shouldn't be stemmed
at query time".

Now if WordDelimiterFilterFactory comes before the stemmer (as it really
should), the $ is removed and the signal to not stem is lost. Oops. And
any of the ReplaceFilterFactories often remove such terms. And....

So the "usual" answer is either to use the KeywordRepeatFilterFactory,
or use a copyField that doesn't stem and when exact matches are
required, search on that field.

Best,
Erick

On Tue, Aug 25, 2015 at 5:05 AM, Modassar Ather <modather1981@gmail.com> wrote:
> Can
> anyone tell me why this option is not provided for Stemming.
>
> I am not sure about it but the original token can be preserved by using
> <filter class="solr.KeywordRepeatFilterFactory"/> too.
> To avoid any duplicate token in the document <filter
> class="solr.RemoveDuplicatesTokenFilterFactory"/> can be used at the end of
> analysis chain.
>
> Hope this helps.
>
> Regards,
> Modassar
>
> On Tue, Aug 25, 2015 at 2:12 PM, Vishnu Mishra <vdilipm@gmail.com> wrote:
>
>> Hi,
>>
>> I was working with Lucene 5.2 and trying to index some document. I am using
>> EnglishMinimalStemFilterFactory and I found that there is no option for
>> keeping the original text as wel as analyzed term into lucene index.
>> WordDelimiterFilterFactory  provides preserveOriginal option to do this.
>> Can
>> anyone tell me why this option is not provided for Stemming. For e.g. if I
>> want to store both *Methods* and *Method* in my index then I think there is
>> no option is available in Lucene to do this.  I also noticed that if we
>> place EnglishMinimalStemFilterFactory after WordDelimiterFilterFactory with
>> option preserveOriginal ="1" then  it store both *Methods* and *Method*.
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://lucene.472066.n3.nabble.com/Preserve-Original-Option-In-Stemming-EnglishMinimalStemFilterFactory-tp4225116.html
>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message