Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of eirikrwu@gmail.com designates
 209.85.223.176 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <A5A0548D-7D1C-458A-BD04-C910595C5642@wunderwood.org>
References: 
 <CAEHBH_OhaBhq4X69W8je0x=3YZv5Xv0psS_S_Tp364MSbkvU0w@mail.gmail.com>
	<6C78E97C707B5B4C8CC61D44F8754586392857@SUEX10-mbx-03.ad.syr.edu>
	<CAK9GPcR1L5jQ7fjoHk8H2e0kBsCRdG6W+3X_NM6bXBrw60L49g@mail.gmail.com>
	<A5A0548D-7D1C-458A-BD04-C910595C5642@wunderwood.org>
Date: Fri, 31 Aug 2012 08:41:55 +0800
Message-ID: 
 <CAEHBH_M0k4mu7d7+mBGARFY-hM1V7h4BWGJNeVuZLDS1ZyZO4w@mail.gmail.com>
Subject: Re: Solr4 distributed IDF
From: Eric Wu <eirikrwu@gmail.com>
To: solr-user@lucene.apache.org
Content-Type: multipart/alternative; boundary=20cf303347bd0cdc1f04c8850fdd

--20cf303347bd0cdc1f04c8850fdd
Content-Type: text/plain; charset=ISO-8859-1

Hi Walter,

    Thank you for your help. I think you are right, the most important
issue here is "the most selective terms are rare". So I probably still need
to implement distributed IDF to get better results.

On Fri, Aug 31, 2012 at 8:36 AM, Walter Underwood <wunder@wunderwood.org>wrote:

> That is true if you randomly distribute the documents. If they are
> distributed according to topic, there can be some big anomalies.
>
> Also, the DFs for rare terms will have bigger errors. There is some
> statistical theorem about this, but I can't remember it right now. Thanks
> to Zipf, most of your terms are rare. Also, the most selective terms are
> rare.
>
> wunder
>
> On Aug 30, 2012, at 5:25 PM, Lance Norskog wrote:
>
> > The math for "confidence values" in probability theory shows that
> > distributed DF does not matter after not very many documents. If you
> > have 10s of thousands of documents in each shard, don't worry.
> >
> > On Thu, Aug 30, 2012 at 1:19 PM, Steven A Rowe <sarowe@syr.edu> wrote:
> >> Hi Ke,
> >>
> >> Have you seen <https://issues.apache.org/jira/browse/SOLR-1632>?
> >>
> >> Steve
> >>
> >> -----Original Message-----
> >> From: Eric Wu [mailto:eirikrwu@gmail.com]
> >> Sent: Thursday, August 30, 2012 3:05 AM
> >> To: solr-user@lucene.apache.org
> >> Subject: Solr4 distributed IDF
> >>
> >> Hi there,
> >>
> >> Does there exist any issue ticket about the distributed IDF feature in
> >> solr4? Or maybe there already have some patches that I can use? Thank
> you
> >> very much.
> >>
> >> --
> >> Ke Wu,
> >> Best Regards
>
>
>
>
>


-- 
Ke Wu,
Best Regards

--20cf303347bd0cdc1f04c8850fdd--