lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jamie Johnson <jej2...@gmail.com>
Subject Re: Solr Cloud Number of Shard Limitation?
Date Thu, 29 Sep 2011 02:35:42 GMT
Thanks Mark found the TODO in ZkStateReader.java

// TODO: - possibly: incremental update rather than reread everything

Was there a patch they provided back to address this?

On Tue, Sep 27, 2011 at 9:20 PM, Mark Miller <markrmiller@gmail.com> wrote:
>
> On Sep 26, 2011, at 11:42 AM, Jamie Johnson wrote:
>
>> Is there any limitation, be it technical or for sanity reasons, on the
>> number of shards that can be part of a solr cloud implementation?
>
>
> The loggly guys ended up hitting a limit somewhere. Essentially, whenever the cloud state
is updated, info is read about each shard to update the state (from zookeeper). There is a
TODO that I put in there that says something like, "consider updating this incrementally"
- usually the data on most shards has not changed, so no reason to read it all. They implemented
that today in their own code, but we have not yet done this in trunk. What that places the
upper limit at, I don't know - I imagine it takes quite a few shards before it ends up being
too much of a problem - they shard by user I believe, so lot's of shards.
>
>
> - Mark Miller
> lucidimagination.com
> 2011.lucene-eurocon.org | Oct 17-20 | Barcelona
>
>
>
>
>
>
>
>
>
>
>

Mime
View raw message