Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of erickerickson@gmail.com
 designates 209.85.216.44 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CALftMhEEKwnWuKzESQD9J5uiMMVp2fxa8Rco0EP5DHge5jJxsA@mail.gmail.com>
References: 
 <CALftMhHmx_ZtyKUxtdYFd47nAj7R8Y761VdiY1cvh-kUQsbONg@mail.gmail.com>
	<CAN4YXveuDgsKSF8yxYUuWk6uyL4sk0NBh_VTn3TiahD0473ABQ@mail.gmail.com>
	<CALftMhFo9S=uvigTHGAKuSrL1KU13hfDuTd7v4q9RY=+jy7W1Q@mail.gmail.com>
	<CAN4YXvfxaOmki3oDs38mx=xVnwdb1AbQtKaexi6Yd85aWvgcNg@mail.gmail.com>
	<CALftMhEEKwnWuKzESQD9J5uiMMVp2fxa8Rco0EP5DHge5jJxsA@mail.gmail.com>
Date: Sun, 23 Jun 2013 10:40:54 -0700
Message-ID: 
 <CAN4YXvcgm_eg_q65YKtL46t_w0_FQEc+nJ218y03-ki8heSbbQ@mail.gmail.com>
Subject: Re: Sharding and Replication
From: Erick Erickson <erickerickson@gmail.com>
To: solr-user@lucene.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Asif:

Thanks, this is great info and may add to the priority of making this
configurable.
I raised a JIRA, see: https://issues.apache.org/jira/browse/SOLR-4956
and feel free to add anything you'd like or correct anything I didn't get
right.

Best
Erick

On Sat, Jun 22, 2013 at 10:16 PM, Asif <tallasif@gmail.com> wrote:
> Erick,
>
> Its a completely practical problem - we are exploring Solr to build a real
> time analytics/data solution for a system handling about 1000 qps. We have
> various metrics that are stored as different collections on the cloud,
> which means very high amount of writes. The cloud also needs to support
> about 300-400 qps.
>
> We initially tested with a single Solr node on a 16 core / 24 GB box  for a
> single metric. We saw that writes were not a issue at all - Solr was
> handling it extremely well. We were also able to achieve about 200 qps from
> a single node.
>
> When we set up the cloud ( a ensemble on 6 boxes), we saw very high CPU
> usage on the replicas. Up to 10 cores were getting used for writes on the
> replicas. Hence my concern with respect to batch updates for the replicas.
>
> BTW, I altered the maxBufferedAddsPerServer to 1000 - and now CPU usage is
> very similar to single node installation.
>
> - Asif
>
>
>
>
>
>
> On Sat, Jun 22, 2013 at 9:53 PM, Erick Erickson <erickerickson@gmail.com>wrote:
>
>> Yeah, there's been talk of making this configurable, but there are
>> more pressing priorities so far.
>>
>> So just to be clear, is this theoretical or practical? I know of several
>> very
>> high-performance situations where 1,000 updates/sec (and I'm assuming
>> that it's 1,000 docs/sec not 1,000 batches of 1,000 docs) hasn't caused
>> problems here. So unless you're actually seeing performance problems
>> as opposed to fearing that there _might_ be, I'd just go on the to the next
>> urgent problem.
>>
>> Best
>> Erick
>>
>> On Fri, Jun 21, 2013 at 8:34 PM, Asif <tallasif@gmail.com> wrote:
>> > Erick,
>> >
>> > Thanks for your reply.
>> >
>> > You are right about 10 updates being batch up - It was hard to figure out
>> > due to large number of updates/logging that happens in our system.
>> >
>> > We are batching 1000 updates every time.
>> >
>> > Here is my observation from leader and replica -
>> >
>> > 1. Leader logs are clearly indicating that 1000 updates arrived - [ (1000
>> > adds)],commit=]
>> > 2. On replica - for each 1000 document adds on leader - I see a lot of
>> > requests on replica - with no indication of how many updates in each
>> > request.
>> >
>> > Digging a little bit into Solr code  I figured this variable I am
>> > interested in - maxBufferedAddsPerServer is set to 10 -
>> >
>> >
>> http://svn.apache.org/viewvc/lucene/dev/trunk/solr/core/src/java/org/apache/solr/update/SolrCmdDistributor.java?view=markup
>> >
>> > This means for a batch update of 1000 documents - we will be seeing 100
>> > requests for replica - which translates into 100 writes per collection
>> per
>> > second in our system.
>> >
>> > Should this variable be made configurable via solrconfig.xml (or any
>> other
>> > appropriate place)?
>> >
>> > A little background about a system we are trying to build - real time
>> > analytics solution using the Solr Cloud + Atomic updates - we have very
>> > high amount of writes - going as high as 1000 updates a second (possibly
>> > more in long run).
>> >
>> > - Asif
>> >
>> >
>> >
>> >
>> >
>> > On Sat, Jun 22, 2013 at 4:21 AM, Erick Erickson <erickerickson@gmail.com
>> >wrote:
>> >
>> >> Update are batched, but it's on a per-request basis. So, if
>> >> you're sending one document at a time you'll won't get any
>> >> batching. If you send 10 docs at a time and they happen to
>> >> go to 10 different shards, you'll get 10 different update
>> >> requests.
>> >>
>> >> If you're sending 1,000 docs per update you' should be seeing
>> >> some batching going on.
>> >>
>> >> bq:  but why not batch them up or give a option to batch N
>> >> updates in either of the above case
>> >>
>> >> I suspect what you're seeing is that you're not sending very
>> >> many docs per update request and so are being mislead.
>> >>
>> >> But that's a guess since you haven't provided much in the
>> >> way of data on _how_ you're updating.
>> >>
>> >> bq: the cloud eventually starts to fail
>> >> How? Details matter.
>> >>
>> >> Best
>> >> Erick
>> >>
>> >> On Wed, Jun 19, 2013 at 4:23 AM, Asif <tallasif@gmail.com> wrote:
>> >> > Hi,
>> >> >
>> >> > I had questions on implementation of Sharding and Replication
>> features of
>> >> > Solr/Cloud.
>> >> >
>> >> > 1. I noticed that when sharding is enabled for a collection -
>> individual
>> >> > requests are sent to each node serving as a shard.
>> >> >
>> >> > 2. Replication too follows above strategy of sending individual
>> documents
>> >> > to the nodes serving as a replica.
>> >> >
>> >> > I am working with a system that requires massive number of writes - I
>> >> have
>> >> > noticed that due to above reason - the cloud eventually starts to fail
>> >> > (Even though I am using a ensemble).
>> >> >
>> >> > I do understand the reason behind individual updates - but why not
>> batch
>> >> > them up or give a option to batch N updates in either of the above
>> case
>> >> - I
>> >> > did come across a presentation that talked about batching 10 updates
>> for
>> >> > replication at least, but I do not think this is the case.
>> >> > - Asif
>> >>
>>