Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of heimann.richard@gmail.com
 designates 74.125.82.48 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=p15YF5AWTW42syZAg3uxTI7c94KAj5id7V0Mr1xPikznAiMYkiJV1jGhO0Z/nfwqNU
         LPRcHRoYiwrekRhUrEy50YGqdVNXpg41IzeJQ+YX5i7EycUhEs/X4pWmpJizmtynnxTT
         OYz1YxerqDeg2iMriRl/6u8ywzs8A1raSJ3w0=
MIME-Version: 1.0
In-Reply-To: <E9DBDB2D-7FFE-47D3-84E0-53E38F2E759A@hoplahup.net>
References: <BANLkTi=2hG=2yZ3sbjedWicCAqJ9WAVj1A@mail.gmail.com>
 <BANLkTik9OEg99E89S1cir9TWSEOnkG1MHQ@mail.gmail.com>
 <E9DBDB2D-7FFE-47D3-84E0-53E38F2E759A@hoplahup.net>
From: Rich Heimann <heimann.richard@gmail.com>
Date: Thu, 19 May 2011 10:20:40 -0400
Message-ID: <BANLkTinLqi+dM2Pk1Yd03GrYNgeg=wAMDg@mail.gmail.com>
Subject: Re: Please help me with a basic question...
To: java-user@lucene.apache.org
Content-Type: multipart/alternative; boundary=001517592688e18ac304a3a1b6e5

--001517592688e18ac304a3a1b6e5
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Thanks Paul,

I do not know what duplicates are in this case and it is the denominator of
the TF that bothers me more than the numerator of the TF (if that is in fac=
t
what you are suggesting).

What have been the effects of ignoring the IDF? When is it appropriate. It
would seem that by doing so rare terms have less (no) weight. Thoughts?

Thanks again,
Rich


On Wed, May 18, 2011 at 3:34 PM, Paul Libbrecht <paul@hoplahup.net> wrote:

> Richard,
>
> in SOLR at least there's an analyzer that avoids duplicates.
> I think that would solve it.
> There's also somewhere the option to ignore IDF (in similarity? in
> solrconfig?).
>
> paul
>
>
> Le 18 mai 2011 =E0 21:30, Rich Heimann a =E9crit :
>
> > Hello all,
> >
> > This is my first time on the list and my first question...forgive me it
> this
> > has been hacked out in the past.
> >
> > We have set up Lucene/Solr and are getting somewhat spurious results. I=
t
> > appears to be a result of heterogeneous document sizes. In other words,
> the
> > top results are sometimes (at least when the user is using typical sear=
ch
> > terms) monopolized by a distinct type of document, which is otherwise
> small
> > (in number of terms). It appears that TF/IDF even with the cosine
> similarity
> > seems to be sensitive to document size. I have run some tests and it in
> fact does
> > appear to be the case.
> >
> > (Number of times the term appears in a document)/(Total Number of terms
> in
> > that document) * Log10(Number of total documents/Number of times search
> term
> > appears in all documents)
> >
> > Are there any suggestions or best practices to deal with the intrinsic
> > heterogeneity in a corpus.
> >
> > Thank you,
> > Rich
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

--001517592688e18ac304a3a1b6e5--