Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of ian.lea@gmail.com designates
 209.85.210.176 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <4ED3EFB4.1050805@digitorial.co.uk>
References: <4ED3DC80.9010606@digitorial.co.uk>
 <6C78E97C707B5B4C8CC61D44F8754586072F8C@SUEX10-mbx-03.ad.syr.edu>
 <4ED3EFB4.1050805@digitorial.co.uk>
From: Ian Lea <ian.lea@gmail.com>
Date: Mon, 28 Nov 2011 20:51:13 +0000
Message-ID: 
 <CAEY5pxWCpUYFHLLbJqHnmqz3cTSevT=+o-cKUNVLFHdRjR7kcA@mail.gmail.com>
Subject: Re: Analysers for newspaper pages...
To: java-user@lucene.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

You can easily use just the CommonGrams stuff from Solr in your pure
lucene project.

There are a couple of useful docs on stop words and common grams et al at

http://www.hathitrust.org/blogs/large-scale-search/slow-queries-and-common-=
words-part-1
http://www.hathitrust.org/blogs/large-scale-search/slow-queries-and-common-=
words-part-2

--
Ian.

On Mon, Nov 28, 2011 at 8:31 PM, Dawn Zo=EB Raison <dawn@digitorial.co.uk> =
wrote:
> Hi Steve,
>
> On 28/11/2011 19:43, Steven A Rowe wrote:
>>
>> I assume that when you refer to "the impact of stop words," you're
>> concerned about query-time performance? =A0You should consider the possi=
bility
>> that performance without removing stop words is good enough that you won=
't
>> have to take any steps to address the issue.
>
> Not to fussed about query-time performance; certainly no-one has complain=
ed
> so far. It's more the sheer number of junk pages we get searching on phra=
ses
> that contain stop words - it can lead to hundreds of thousands of results=
,
> and the pedants among our userbase insist on paging through the lot :-|
>
> I'd much rather contain the stop words using a *gram based approach and
> offer a less populous but more accurate resultset.
>
>>
>> That said, there are two filters in Solr 3.X[1] that would do the
>> equivalent of what you have outlined:
>> CommonGramsFilter<http://lucene.apache.org/solr/api/org/apache/solr/anal=
ysis/CommonGramsFilter.html>
>> =A0and
>> CommonGramsQueryFilter<http://lucene.apache.org/solr/api/org/apache/solr=
/analysis/CommonGramsQueryFilter.html>.
>
> We use lucene directly, but I'll take a look - Thanks.
>
>> You can use these filters with a Lucene 3.X application by including the
>> (same-versioned) solr-core jar as a dependency.
>>
>> Steve
>
> --
>
> Rgds.
> *Dawn Raison*
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org