Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of greg.pendlebury@gmail.com
 designates 209.85.212.178 as permitted sender)
MIME-Version: 1.0
Date: Wed, 4 Jun 2014 13:32:33 +1000
Message-ID: 
 <CA+A4HB2kngRB3ix_3x1pa8JvRSFssWWurFYojHcPFxMTHtHjOA@mail.gmail.com>
Subject: SolrCloud leaders using more disk space
From: Greg Pendlebury <greg.pendlebury@gmail.com>
To: solr-user@lucene.apache.org
Content-Type: multipart/alternative; boundary=047d7b3a817661476e04fafa47e9

--047d7b3a817661476e04fafa47e9
Content-Type: text/plain; charset=UTF-8

Hi all,

We launched our new production instance of SolrCloud last week and since
then have noticed a trend with regards to disk usage. The non-leader
replicas all seem to be self-optimizing their index segments as expected,
but the leaders have (on average) around 33% more data on disk. My
assumption is that leader's are not self-optimising (or not to the same
extent)... but it is still early days of course.

If it helps, there are 45 JVMs in the cloud, with 15 shards and 3 replicas
per shard. Each non-leader shard is sitting at between 59GB and 87GB on
their SSD, but the leaders are between 84GB and 116GB.

We have pretty much constant read and write traffic 24x7, with just 'slow'
periods overnight when write traffic is < 1 document per second and
searches are between 1 and 2 per second. Is this light level of traffic
still too much for the leaders to self-optimise?

I'd also be curious to hear about what others are doing in terms of
operating procedures. We load test before launch what would happen if we
turned off JVMs and forced recovery events. I know that these things all
work, just that customers will experience slower search responses whilst
they occur. For example, a restore from a leader to a replica under load
testing for us takes around 30 minutes and response times drop from around
200-300ms average to 1.5s average.

Bottleneck appears to be network I/O on the servers. We haven't explored
whether this is specific to the servers replicating, or saturation of the
of the infrastructure that all the servers share, because...

This performance is acceptable for us, but I'm not sure if I'd like to
force that event to occur unless required... this is following the line of
reasoning proposed internally that we should periodically rotate leaders by
turning them off briefly. We aren't going to do that unless we have a
strong reason though. Does anyone try to manipulate production instances
that way?

Vaguely related to this is leader distribution. We have 9 physical servers
and 5 JVMs running on each server. By virtue of the deployment procedures
the first 3 servers to come online are all running 5 leaders each. Is there
any merit in 'moving' these around (by reboots)?

Our planning up to launch was based on lots of mailing list response we'd
seen that indicated leaders had no significant performance difference to
normal replicas, and all of our testing has agreed with that. The disk size
'issue' (which we aren't worried about... yet. It hasn't been in prod long
enough to know for certain) may be the only thing we've seen so far.

Ta,
Greg

--047d7b3a817661476e04fafa47e9--