Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
From: =?utf-8?Q?Emir_Arnautovi=C4=87?= <emir.arnautovic@sematext.com>
Content-Type: multipart/alternative;
 boundary="Apple-Mail=_1B84B2A5-F987-4ABB-8DC7-38BB445EECB6"
Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\))
Subject: Re: Advice on Stemming in Solr
Date: Thu, 2 Nov 2017 10:46:35 +0100
References: <CAF2DzVXO4Ke4hWVZrkwuyJNXmjtZGdCruwkWVyVP8W5Ccf_QPA@mail.gmail.com>
 <7F86BE83-5347-456A-B457-5FE000E51C69@sematext.com>
 <CAF2DzVUWFRGaATjtYmfASc5pgYOhmMYg9MUTnDb2arGo29p=hQ@mail.gmail.com>
To: solr-user@lucene.apache.org
In-Reply-To: <CAF2DzVUWFRGaATjtYmfASc5pgYOhmMYg9MUTnDb2arGo29p=hQ@mail.gmail.com>
Message-Id: <CFEF8BE5-26DE-4DA6-8775-6BEFDDF5A706@sematext.com>
archived-at: Thu, 02 Nov 2017 09:46:50 -0000

--Apple-Mail=_1B84B2A5-F987-4ABB-8DC7-38BB445EECB6
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8

Hi Edwin,
It seems that it would be best if you do not apply *ing stemming rule at =
all. The first idea is to trick stemmer and replace any word that ends =
with ing to some nonexisting char combination e.g. =E2=80=98wqx=E2=80=99. =
You can use solr.PatternReplaceFilterFactory to do that. You can switch =
it back after stemming if want to have proper token in index.

HTH,
Emir=20
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/


> On 2 Nov 2017, at 03:23, Zheng Lin Edwin Yeo <edwinyeozl@gmail.com> =
wrote:
>=20
> Hi Emir,
>=20
> We do have quite alot of words that should not be stemmed. Currently, =
the
> KStemFilterFactory are stemming all the non-English words that end =
with
> "ing" as well. There are quite alot of places and names which ends in
> "ing", and all these are being stemmed as well, which leads to an
> inaccurate search.
>=20
> Regards,
> Edwin
>=20
>=20
> On 1 November 2017 at 18:20, Emir Arnautovi=C4=87 =
<emir.arnautovic@sematext.com>
> wrote:
>=20
>> Hi Edwin,
>> If the number of words that should not be stemmed is not high you =
could
>> use KeywordMarkerFilterFactory to flag those words as keywords and it
>> should prevent stemmer from changing them.
>> Depending on what you want to achieve, you might not be able to avoid
>> using stemmer at indexing time. If you want to find documents that =
contain
>> only =E2=80=9Cwalking=E2=80=9D with search term =E2=80=9Cwalk=E2=80=9D,=
 then you have to stem at index
>> time. Cases when you use stemming on query time only are rare and =
specific.
>> If you want to prefer exact matches over stemmed matches, you have to
>> index same content with and without stemming and boost matches on =
field
>> without stemming.
>>=20
>> HTH,
>> Emir
>> --
>> Monitoring - Log Management - Alerting - Anomaly Detection
>> Solr & Elasticsearch Consulting Support Training - =
http://sematext.com/
>>=20
>>=20
>>=20
>>> On 1 Nov 2017, at 10:11, Zheng Lin Edwin Yeo <edwinyeozl@gmail.com>
>> wrote:
>>>=20
>>> Hi,
>>>=20
>>> We are currently using KStemFilterFactory in Solr, but we found that =
it
>> is
>>> actually doing stemming on non-English words like "ximenting", which =
it
>>> stem to "ximent". This is not what we wanted.
>>>=20
>>> Another option is to use the HunspellStemFilterFactory, but there =
are
>> some
>>> English words like "running", walking" that are not being stemmed.
>>>=20
>>> Would like to check, is it advisable to use Stemming at index? Or we
>> should
>>> not use Stemming at index time, but at query time, do a search for =
the
>>> stemmed words as well, like for example, if the user search for
>> "walking",
>>> we will do the search together with "walk", and the actual word of
>> walking
>>> will have higher weightage.
>>>=20
>>> I'm currently using Solr 6.5.1.
>>>=20
>>> Regards,
>>> Edwin
>>=20
>>=20


--Apple-Mail=_1B84B2A5-F987-4ABB-8DC7-38BB445EECB6--