Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of cdoronc@gmail.com designates
 209.85.216.48 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=avAamPh+uDdopqOraDJEvLIYxXCb8KmeLiUw6YyhaxKrT4ExX+JuuFR9O2GmOPYExe
         Gdv7z471HRSuSd7iMdhRwCIPL9CU/5IE+Fc71DnDF+yvp7QqOYOR1ETG2FAmLQMQfRFH
         OzUSdPlUHx9h6UkyJQ+Am/8zu2BxhZM2pU7K4=
MIME-Version: 1.0
In-Reply-To: 
 <E1E61A9983E3F94DAA95AEDE2D487E700618C80500@mbx2.hostedexchange.local>
References: 
 <E1E61A9983E3F94DAA95AEDE2D487E700618C7FE85@mbx2.hostedexchange.local>
	<AANLkTimqVLFrH1FGACaE3iTGidvJuEvY-ptdw4jbMuDY@mail.gmail.com>
	<E1E61A9983E3F94DAA95AEDE2D487E700618C80500@mbx2.hostedexchange.local>
Date: Tue, 11 Jan 2011 00:16:22 +0200
Message-ID: <AANLkTinv7jjzxKDq4=sVUTbf+6k1mqz1UwHZV1V2wKqQ@mail.gmail.com>
Subject: Re: Creating an index with multiple values for a single field
From: Doron Cohen <cdoronc@gmail.com>
To: java-user@lucene.apache.org
Content-Type: multipart/alternative; boundary=0016e646514873fe400499855172

--0016e646514873fe400499855172
Content-Type: text/plain; charset=ISO-8859-1

On Mon, Jan 10, 2011 at 7:44 PM, Ryan Aylward <ryan@glassdoor.com> wrote:

> We do leverage synonyms but they are not appropriate for this case. We use
> synonyms for words that are truly synonymous for the entire index such as
> "inc" and "incorporated". Those words are always interchangeable. However,
> many of the employer alternate names are only valid for a single employer
> not for the entire index.
> We do disable the lengthNorm but we benefit from tf and idf so disabling
> those would cause more harm than good.
> Any other suggestions would be appreciated.
>

How about indexing this specific field without analysis - except perhaps for
lower casing - i.e. in the above example the field would have exactly 3
tokens: [wal-mart], [wal-mart stores], [walmart]. At search time this field
would be treated the same way, that is, no analysis except for lower casing.
Since norms are already omitted for this field its lengths  differences
between docs would not affect scores.
HTH, Doron


> Thanks.
>
> -----Original Message-----
> From: Anshum [mailto:anshumg@gmail.com]
> Sent: Friday, January 07, 2011 7:38 PM
> To: java-user@lucene.apache.org
> Subject: Re: Creating an index with multiple values for a single field
>
> Hi Ryan,
> You should try the synonym filter. That should help you with this kinda
> problem.
> You could also look at turning off norms for the name field, or turning off
> tf or idf.
>
> --
> Anshum Gupta
> http://ai-cafe.blogspot.com
>
>
> On Sat, Jan 8, 2011 at 6:03 AM, Ryan Aylward <ryan@glassdoor.com> wrote:
>
> > Our business has a need to allow for multiple values for a single field.
> > For example, we have an index of employers where an employer often has
> > multiple ways people refer to it. For example, the company "Wal-mart" is
> > referred to as:
> >
> > 1)      Wal-mart
> >
> > 2)      Wal-mart Stores
> >
> > 3)      Walmart
> > I would like a search for any of these 3 terms to match the Wal-mart
> > employer.
> >
> > I've tried two different approaches for this.
> >
> > Approach 1: Create multiple values for the same field. So the document
> has
> > these three fields:
> >
> > 1)      name=Wal-mart
> >
> > 2)      name=Wal-mart Stores
> >
> > 3)      name=Walmart
> > The problem with this is Lucene seems to treat the 3 different fields as
> > one long field of "Wal-mart Wal-mart Stores Walmart". This is problematic
> > b/c term frequencies is 2 when a user searches for "Wal-mart".
> >
> > Approach 2: Create different named fields for each value so the document
> > has these 3 fields:
> >
> > 1)      name1=Wal-mart
> >
> > 2)      name2=Wal-mart Stores
> >
> > 3)      name3=Walmart
> > This fixes the issue above but introduces a different problem. The idf
> > calculation is incorrect b/c idf is calculated per field. Most employers
> > only have one name or maybe 2 names. So the name3 fields idf ends up
> being
> > much higher b/c there are fewer docs with a given term in the name3
> field.
> >
> > For now, I'm going with approach 2 but overriding the IndexReader.
> > IndexReader.docFreq(Term t) method always returns the doc frequency from
> the
> > name1 field even if the Term t is actually for name2 or name3, etc. But
> this
> > doesn't feel like a clean solution.
> >
> > Any suggestions on how to deal with this? Any ideas would be appreciated.
> > Ryan Aylward
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

--0016e646514873fe400499855172--