Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of simon.willnauer@googlemail.com
 designates 209.85.220.225 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:reply-to:in-reply-to:references:date:message-id
         :subject:from:to:content-type:content-transfer-encoding;
        b=WgbZpxpxl7ezjTyyKKUzuNrjgW+C0U4WOSk5G4x3Gu2NRKvRhDxRn1wJ8qkqFyo4pb
         UTeWI8pBjXCTQqBSAOGJAa4HE7OyJw/E9MSRIza0OPgfQDexXdkmDozWHnBe9qhyPguo
         A4zsglZ8mEjwXf49l1Jj/eAsgMmoj5rxK1044=
MIME-Version: 1.0
Reply-To: simon.willnauer@gmail.com
In-Reply-To: <20090804170013.6d9b5753@pc-4176.kl.dfki.de>
References: <20090804115016.64b58dfe@pc-4176.kl.dfki.de>
	 <180060.75297.qm@web50309.mail.re2.yahoo.com>
	 <20090804170013.6d9b5753@pc-4176.kl.dfki.de>
Date: Wed, 5 Aug 2009 09:58:45 +0200
Message-ID: <f18c9dde0908050058j57c3a790u27a0b16c73625837@mail.gmail.com>
Subject: Re: ParallelMultiSearcher and idf
From: Simon Willnauer <simon.willnauer@googlemail.com>
To: java-user@lucene.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hey Christian,

you might wanna look at distributed solr
(http://wiki.apache.org/solr/DistributedSearch) or if you haven't done
so have a look at the Katta project
(http://katta.sourceforge.net/documentation/how-katta-works) maybe
this can help you out.
About distributed IDF and Scoring have a look at this link:
http://wunderwood.org/most_casual_observer/2007/04/progressive_reranking.ht=
ml

Simon

On Tue, Aug 4, 2009 at 5:00 PM, Christian
Reuschling<christian.reuschling@gmail.com> wrote:
> Hi Otis,
>
> thanks for the answer - I'm aware of Solr, but it seems this is - accordi=
ng to
> its abstraction level - too generalized for us. Solr seems to be nice in =
the
> case you want to use the black box, and won't be aware of 'what is under =
the
> hood'.
> But maybe I'm totaly wrong. At least, it would be from interest how Solr
> realizes its distributed search, in the case it makes something different
> than using the core-Lucene ParallelMultiSearcher with RemoteSearchables. =
Maybe
> on this list somebody knows the answer.
>
>
>
>
>
> On Tue, 4 Aug 2009 07:20:23 -0700 (PDT)
> Otis Gospodnetic <otis_gospodnetic@yahoo.com> wrote:
>
>> Hi Christian,
>>
>> You didn't mention Solr, so I'm not sure if you are aware of it. =C2=A0M=
aybe Solr
>> meets your needs?
>>
>> =C2=A0Otis
>> --
>> Sematext is hiring -- http://sematext.com/about/jobs.html?mls
>> Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
>>
>>
>>
>> ----- Original Message ----
>> > From: Christian Reuschling <christian.reuschling@gmail.com>
>> > To: java-user@lucene.apache.org
>> > Sent: Tuesday, August 4, 2009 5:50:16 AM
>> > Subject: ParallelMultiSearcher and idf
>> >
>> > Hello,
>> >
>> > when searching over multiple indices, we create one IndexReader for ea=
ch
>> > index, and wrap them into a MultiReader, that we use for IndexSearcher
>> > creation.
>> >
>> > This is fine for searching multiple indices on one machine, but in the=
 case
>> > the indices are distributed over the (intra)net, this scenario has sev=
eral
>> > lacks:
>> >
>> > - searching/scoring/sorting is 100% on the client machine, so you need=
 all
>> > the ram and cpu power at every client.
>> > - all the data necessary for scoring must go over the net - so the tra=
ffic
>> > =C2=A0 should be significantly higher
>> > - thus, there is a lack of overall performance
>> >
>> > Nevertheless, creating a MultiReader and making a searcher out of it h=
as one
>> > advantage (at least can be an advantage depending on the scenario): Th=
e
>> > document freqiencies of a term will be summed up, and thus it is 100%
>> > transparent for scoring whether the indices are splittet or not.
>> >
>> > I'm wondering whether there is the possibility to get the advantages o=
f both
>> > scenarios, e.g. by first summing up the query terms-related document
>> > frequencies, and sending them together with the query to every
>> > (remote)searcher of ParallelMultiSearcher, for scoring.
>> >
>> > Maybe this is exactly what ParallelMultiSearcher does, and I haven't s=
een
>> > it?
>> >
>> >
>> > Thanks for clarification!
>> >
>> > Chris
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org