Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of alxcwll@gmail.com designates
 209.85.213.48 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=jEvEZ2nebxA1FEPIRnnOFHW7gF39QgIDWS3nhkZyDZfKW90PHinv/1ZCs00iH1OAe1
         JFowPjkyeQNVvg12NkxDaBiN/zV433sudi2qN804zdFb0qSDlLj/x0nLdqWJhd1X3c7G
         sK/sS0tG06CLKtiGdm76dzye+4zDnxRoEcYn8=
MIME-Version: 1.0
In-Reply-To: <AANLkTikYft0Xozy1w=D8MLbyJ3-jwxofXF7gt5hEjXF=@mail.gmail.com>
References: <1296059354.3260.19.camel@soheb-1201N>
	<AANLkTi=_ZfdqTJtRCUcT4KgrZ8dJB=2-2RDJMbqTUaVx@mail.gmail.com>
	<1296333246.2931.12.camel@soheb-1201N>
	<AANLkTinYmYAuCFyTx_1KmcHK6yjudd8OP3Xf_ce-8ePK@mail.gmail.com>
	<1296345418.18365.1417931133@webmail.messagingengine.com>
	<AANLkTi=csrrEhmTEYf=7eo9bSFF+eYz4ZqY3K7xFBO1i@mail.gmail.com>
	<1296559676.20400.1418327987@webmail.messagingengine.com>
	<AANLkTikjhmho8_F27pGOBJdaPYMiyph79OsoExzFgtFa@mail.gmail.com>
	<AANLkTi=t7M5ecV+AC_Fha98wdO6brY5EEtrc4r6gWoPH@mail.gmail.com>
	<AANLkTi=hNYPHyLS3oycG8r7yZiY1=mgKnKxExyFfm4Jn@mail.gmail.com>
	<1296592736.25523.1418427511@webmail.messagingengine.com>
	<AANLkTikj10Q-E4zm=YA=gGeucdyj3+h70133+P1vBDr_@mail.gmail.com>
	<1296635467.5878.1418514861@webmail.messagingengine.com>
	<AANLkTimqBQFFxtu63VykVq_Pb+affRHw6XVpU+8T5pae@mail.gmail.com>
	<1296752436.28572.1418784405@webmail.messagingengine.com>
	<AANLkTimQhwFva1eEiRCHPuSvomgELmsh=m=WBcan=SNY@mail.gmail.com>
	<AANLkTinwXvYZhr+1Tm+GFLvZBXkNFB=6TOOr+D-h74HA@mail.gmail.com>
	<1297086938.29667.1419359807@webmail.messagingengine.com>
	<AANLkTikYft0Xozy1w=D8MLbyJ3-jwxofXF7gt5hEjXF=@mail.gmail.com>
Date: Mon, 14 Feb 2011 15:04:21 +0000
Message-ID: <AANLkTi=SoQOfBfr0E034qnAOHNVVLjTOb-rFgX38thX6@mail.gmail.com>
Subject: Re: Distributed Indexing
From: Alex Cowell <alxcwll@gmail.com>
To: dev@lucene.apache.org
Content-Type: multipart/alternative; boundary=20cf3054a15ddc533a049c3f5c74

--20cf3054a15ddc533a049c3f5c74
Content-Type: text/plain; charset=ISO-8859-1

I've uploaded a patch of what we've done so far:

https://issues.apache.org/jira/browse/SOLR-2358

It's still very much work in progress and there are some obvious issues
which are being resolved at the moment (such as the inefficient method of
waiting for all the docs to be processed before distributing them in one
batch and handling shard replicas), but any feedback is welcomed.

As it stands, you can distribute add and commit requests using the
HashedDistributionPolicy by simply specifying a 'shards' request parameter.
Using a user specified distribution policy (either as a param in the URL or
defined in the solrconfig as Upayavira suggested) is in the works as well.
Regarding that, I figure the priority for determining which policy to use
would be (highest to lowest):

1. Param in the URL
2. Specified in the solrconfig
3. Hard-coded default to fall back on

That way if a user changed their mind about which distribution policy they
wanted to use, they could override the default policy with their chosen one
as a request parameter.

The code has only been acceptance tested at the moment. There is a test
class but it's a bit messy, so once that's tidied up and improved a little
more I'll include it in the next patch.


> I haven't had time to follow all of this discussion, but this issue might
> help:
> https://issues.apache.org/jira/browse/SOLR-2355
>

Thanks - very interesting! It's reassuring to see our implementation has
been following a similar structure.

There seem to be some nuances which we have yet to encounter/discover like
the way you've implemented the processCommit() method to wait for all the
adds/deletes to complete before sending the commits. Are these things which
you were aware of in advance that would need to be dealt with?

Alex

--20cf3054a15ddc533a049c3f5c74
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I&#39;ve uploaded a patch of what we&#39;ve done so far:<br><br><a href=3D"=
https://issues.apache.org/jira/browse/SOLR-2358">https://issues.apache.org/=
jira/browse/SOLR-2358</a><br><br>It&#39;s still very much work in progress =
and there are some obvious issues which are being resolved at the moment (s=
uch as the inefficient method of waiting for all the docs to be processed b=
efore distributing them in one batch and handling shard replicas), but any =
feedback is welcomed.<br>

<br>As it stands, you can distribute add and commit requests using the Hash=
edDistributionPolicy by simply specifying a &#39;shards&#39; request parame=
ter. Using a user specified distribution policy (either as a param in the U=
RL or defined in the solrconfig as Upayavira suggested) is in the works as =
well. Regarding that, I figure the priority for determining which policy to=
 use would be (highest to lowest):<br>

<br>1. Param in the URL<br>2. Specified in the solrconfig<br>3. Hard-coded =
default to fall back on<br><br>That way if a user changed their mind about =
which distribution policy they wanted to use, they could override the defau=
lt policy with their chosen one as a request parameter.<br>

<br>The code has only been acceptance tested at the moment. There is a test=
 class but it&#39;s a bit messy, so once that&#39;s tidied up and improved =
a little more I&#39;ll include it in the next patch.<br>=A0<br><blockquote =
style=3D"margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 20=
4); padding-left: 1ex;" class=3D"gmail_quote">

I haven&#39;t had time to follow all of this discussion, but this issue mig=
ht help:<br><a href=3D"https://issues.apache.org/jira/browse/SOLR-2355" tar=
get=3D"_blank">https://issues.apache.org/jira/browse/SOLR-2355</a><br></blo=
ckquote>

<div><br>Thanks - very interesting! It&#39;s reassuring to see our implemen=
tation has been following a similar structure.<br><br>There seem to be some=
 nuances which we have yet to encounter/discover like the way you&#39;ve im=
plemented the processCommit() method to wait for all the adds/deletes to co=
mplete before sending the commits. Are these things which you were aware of=
 in advance that would need to be dealt with?<br>

<br>Alex<br></div>

--20cf3054a15ddc533a049c3f5c74--