Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of erickerickson@gmail.com
 designates 209.85.211.194 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=uu9f0ep05pcUlsKKtlaJugThLl0wTSNrXzkNJoN4YwTG5T0/e/9DifSIyjYkbMdP+Q
         UoB74qSCcgJlOOeyXs/egfnRnQ/GcOd2oLz7E9qtMrEXgSIvGuDACw3fcZhwBawHrXsl
         Jtnwtn51x6pfYAEEZ+nsW/mpmKaL/+PhHFCKo=
MIME-Version: 1.0
In-Reply-To: <DBB93B72D5294D6FA29E665F1DF3D8E4@Rissos>
References: <212353.47457.qm@web113304.mail.gq1.yahoo.com>
	 <DBB93B72D5294D6FA29E665F1DF3D8E4@Rissos>
Date: Fri, 21 May 2010 09:18:54 -0400
Message-ID: <AANLkTimzdUvlLL_EENmhGTF-cCHmJhcKIgI47udN33CX@mail.gmail.com>
Subject: Re: Stemming and Wildcard Queries
From: Erick Erickson <erickerickson@gmail.com>
To: java-user@lucene.apache.org
Content-Type: multipart/alternative; boundary=000e0cd6afdc70608c04871a8831

--000e0cd6afdc70608c04871a8831
Content-Type: text/plain; charset=ISO-8859-1

Another approach to stemming at index time but still providing exact matches
when requested is to index the stemmed version AND the original version at
the same position (think synonyms). But here's the trick, index the original
token with a special character. For instance, indexing "running" would look
like indexing "run" and "running$". Now, whenever you want the exact match,
just add the "$" to the end of the token.

With this approach, you have to watch that your analyzers don't strip the
'$'...

Of course, each approach has its trade-offs, and the characteristics of your
particular problem may determine which is preferable...

FWIW
Erick

On Thu, May 20, 2010 at 4:48 PM, Herbert Roitblat <herb@orcatec.com> wrote:

> At a general level, we have found that stemming during indexing is not
> advisable.  Sometimes users want the exact form and if you have removed the
> exact form during indexing, obviously, you cannot provide that.  Rather, we
> have found that stemming during search is more useful, or maybe it should be
> called anti-stemming.  For any given input for which the user wants to stem,
> we could derive the variations during the query processing.  E.g., plan can
> be expanded to include plans, planning, planned, etc.
>
> In our application we provide a feature that is sometimes called a word
> wheel.  When someone enters plan in this tool, we show all of the words in
> the index that start with plan. Here are some of the related words:
> plan
> plane
> planes
> planet
> planificaci
> planned
> plannedoutages.xls
> planner
> planners
>
> Just a thought.
> Herb
>
> ----- Original Message ----- From: "Ivan Provalov" <iprovalo@yahoo.com>
> To: <java-user@lucene.apache.org>
> Sent: Thursday, May 20, 2010 1:16 PM
> Subject: Stemming and Wildcard Queries
>
>
>
>  Is there a good way to combine the wildcard queries and stemming?
>>
>> As is, the field which is stemmed at index time, won't work with some
>> wildcard queries.
>>
>> We were thinking to create two separate index fields - one stemmed, one
>> non-stemmed, but we are having issues with our SpanNear queries (they
>> require the same field).
>>
>> We thought to try combining the stemmed and non-stemmed terms in the same
>> field, but we are concerned about the stats being skewed as a result of this
>> (especially for the TermVector stats).  Can overloading the non-stemmed
>> field with stemmed terms cause any issues with the TermVector?
>>
>> Any suggestions?
>>
>> Ivan Provalov
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

--000e0cd6afdc70608c04871a8831--