Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: local policy)
Content-Type: text/plain; charset=iso-8859-1
Mime-Version: 1.0 (Apple Message framework v1078)
Subject: Re: preside != president
From: =?iso-8859-1?Q?Jan_H=F8ydahl_/_Cominvent?= <jan.asf@cominvent.com>
In-Reply-To: <AANLkTimRrU6fARHp58CI8FUnlbh6U2U7zBiF3x0GGa4z@mail.gmail.com>
Date: Mon, 28 Jun 2010 19:54:22 +0200
Content-Transfer-Encoding: quoted-printable
Message-Id: <F2E2E836-C7FA-4E6B-948C-52EE34F63FFE@cominvent.com>
References: <1277726685.6747.2.camel@kratos>
 <9F5FCD40-C9BB-4CFB-BB0D-D3CDF168099A@gmail.com>
 <9eb24a79bbfe195513fa05e0ce2c654c.squirrel@sm.webmail.pair.com>
 <AANLkTimRrU6fARHp58CI8FUnlbh6U2U7zBiF3x0GGa4z@mail.gmail.com>
To: solr-user@lucene.apache.org

Hi,

You might also want to check out the new Lucene-Hunspell stemmer at =
http://code.google.com/p/lucene-hunspell/
It uses OpenOffice dictionaries with known stems in combination with a =
large set of language specific rules.
It handles your example, but it is an early release, so test it =
thoroughly before deploying in production :)

--
Jan H=F8ydahl, search solution architect
Cominvent AS - www.cominvent.com
Training in Europe - www.solrtraining.com

On 28. juni 2010, at 17.43, Joe Calderon wrote:

> the general consensus among people who run into the problem you have
> is to use a plurals only stemmer, a synonyms file or a combination of
> both (for irregular nouns etc)
>=20
> if you search the archives you can find info on a plurals stemmer
>=20
> On Mon, Jun 28, 2010 at 6:49 AM,  <darren@ontrenet.com> wrote:
>> Thanks for the tip. Yeah, I think the stemming confounds search =
results as
>> it stands (porter stemmer).
>>=20
>> I was also thinking of using my dictionary of 500,000 words with =
their
>> complete morphologies and conjugations and create a synonyms.txt to
>> provide english accurate morphology.
>>=20
>> Is this a good idea?
>>=20
>> Darren
>>=20
>>> Hi Darren,
>>>=20
>>> You might want to look at the KStemmer
>>> (http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters/Kstem)
>>> instead of the standard PorterStemmer. It essentially has a =
'dictionary'
>>> of exception words where stemming stops if found, so in your case
>>> president won't be stemmed any further than president (but =
presidents will
>>> be stemmed to president). You will have to integrate it into solr
>>> yourself, but that's straightforward.
>>>=20
>>> HTH
>>> Brendan
>>>=20
>>>=20
>>> On Jun 28, 2010, at 8:04 AM, Darren Govoni wrote:
>>>=20
>>>> Hi,
>>>>  It seems to me that because the stemming does not produce
>>>> grammatically correct stems in many of the cases,
>>>> search anomalies can occur like the one I am seeing where I have a
>>>> document with "president" in it and it is returned
>>>> when I search for "preside", a different word entirely.
>>>>=20
>>>> Is this correct or acceptable behavior? Previous discussions here =
on
>>>> stemming, I was told its ok as long as all the words reduce
>>>> to the same stem, but when different words reduce to the same stem =
it
>>>> seems to affect search results in a "bad way".
>>>>=20
>>>> Darren
>>>=20
>>>=20
>>=20
>>=20