lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Troullis <cptroul...@gmail.com>
Subject Re: Multiple collections vs multiple shards for multitenancy
Date Sat, 06 May 2017 23:26:38 GMT
Hi Erick,

Thanks for the reply, I really appreciate it.

To answer your questions, we have a little over 300 tenants, and a couple
of different collections, the largest of which has ~11 million documents
(so not terribly large). We are currently running standard Solr with simple
master/slave replication, so all of the documents are in a single solr
core. We are planning to move to Solr cloud for various reasons, and as
discussed previously, I am trying to find the best way to distribute the
documents to serve a more NRT focused search case.

I totally get your point on pushing back on NRT requirements, and I have
done so for as long as I can. Currently our auto softcommit is set to 1
minute and we are able to achieve great query times with autowarming.
Unfortunately, due to the nature of our application, our customers expect
any changes they make to be visible almost immediately in search, and we
have recently been getting a lot of complaints in this area, leading to an
initiative to drive down the time it takes for documents to become visible
in search. Which leaves me where I am now, trying to find the right balance
between document visibility and reasonable, stable, query times.

Regarding autowarming, our autowarming times aren't too crazy. We are
warming a max of 100 entries from the filter cache and it takes around 5-10
seconds to complete on average. I suspect our biggest slow down during
autowarming is the static warming query that we have that runs 10+ facets
over the entire index. Our searches are very facet intensive, we use the
JSON facet API to do some decently complex faceting (block joins, etc), and
for whatever reason, even though we use doc values for all of our facet
fields, simply warming the filter cache doesn't seem to prevent a giant
drop off in performance whenever a new searcher is opened. The only way I
could find to prevent the giant performance dropoff was to run a static
warming query on new searcher that runs all of our facets over the whole
index. I haven't found a good way of telling how long this takes, as the
JMX hooks for monitoring autowarming times don't seem to include static
warming queries (that I can tell).

Through experimentation I've found that by sharding my index in Solr cloud,
I can pretty much eliminate autowarming entirely and still achieve
reasonable query times once the shards reach a small enough size (around 1
million docs per shard). This is great, however, your assumptions as to our
tenant size distribution was spot on. Because of this, sharding naturally
using the composite id router with the tenant ID as the key yields an
uneven distribution of documents across shards. Basically whatever unlucky
small tenants happen to get stuck on the same shard as a gigantic tenant
will suffer in performance because of it. That's why I was exploring the
idea of having a tenant per collection or per shard, as a way of isolating
tenants from a performance perspective.

I am a little confused on this statement you made:

> Plus you can't commit
> individually, a commit on one will _still_ commit on all so you're
> right back where you started.


We don't commit manually at all, we rely on auto softcommit to commit for
us. My understanding was that since a shard is basically it's own solr core
under the covers, that indexing a document to a single shard would only
open a new searcher (and thus invalidate caches) on that one shard, and
thus separating tenants in their own shards would mean that tenant A (shard
A) updating it's documents would not require tenant B (shard B) to have all
of it's caches invalidated. Is this not correct?

I'm also not sure I understand what you are saying regarding the hybrid
approach you mentioned. You say to experiment with how many documents are
ideal for a collection, but isn't the number of documents per shard the
more meaningful number WRT performance? I apologize if I am being dense,
maybe I'm not 100% clear on my terminology. My understanding was that a
collection is a logical abstraction consisting of multiple shards/replicas,
and that the shards/replicas were actual physical solr cores. So for
example, what is the difference between having 1000 collections with 1
shard each, vs 1 collection with 1000 shards? Both cases will end up with
the same amount of physical solr cores right? Or am I completely off base?

Thanks again,

Chris

On Sat, May 6, 2017 at 10:36 AM, Erick Erickson <erickerickson@gmail.com>
wrote:

> Well, it's not either/or. And you haven't said how many tenants we're
> talking about here. Solr startup times for a single _instance_ of Solr
> when there are thousands of collections can be slow.
>
> But note what I am talking about here: A single Solr on a single node
> where there are hundreds and hundreds of collections (or replicas for
> that matter). I know of very large installations with 100s of
> thousands of _replicas_ that run. Admittedly with a lot of care and
> feeding...
>
> Sharding a single large collection and using custom routing to push
> tenants to a single shard will be an administrative problem for you.
> I'm assuming you have the typical multi-tenant problems, a bunch of
> tenants have around N docs, some smaller percentage have 3N and a few
> have 100N. Now you're having to keep track of how many docs are on
> each shard, do the routing yourself, etc. Plus you can't commit
> individually, a commit on one will _still_ commit on all so you're
> right back where you started.
>
> I've seen people use a hybrid approach: experiment with how many
> _documents_ you can have in a collection (however you partition that
> up) and use the multi-tenant approach. So you have N collections and
> each collection has a (varying) number of tenants. This also tends to
> flatten out the update process on the assumption that your smaller
> tenants also don't update their data as often.
>
> However, I really have to question one of your basic statements:
>
> "This works fine with aggressive autowarming, but I have a need to reduce
> my NRT
> search capabilities to seconds as opposed to the minutes it is at now,"...
>
> The implication here is that your autowarming takes minutes. Very
> often people severely overdo the warmup by setting their autowarm
> counts to 100s or 1000s. This is rarely necessary, especially if you
> use docValues fields appropriately. Very often much of autowarming is
> "uninverting" fields (look in your Solr log). Essentially for any
> field you see this, use docValues and loading will be much faster.
>
> You also haven't said how many documents you have in a shard at
> present. This is actually the metric I use most often to size
> hardware. I claim you can find a sweet spot where minimal autowarming
> will give you good enough performance, and that number is what you
> should design to. Of course YMMV.
>
> Finally: push back really hard on how aggressive NRT support needs to
> be. Often "requirements" like this are made without much thought as
> "faster is better, let's make it 1 second!". There are situations
> where that's true, but it comes at a cost. Users may be better served
> by a predictable but fast system than one that's fast but
> unpredictable. "Documents may take up to 5 minutes to appear and
> searches will usually take less than a second" is nice and concise. I
> have my expectations. "Documents are searchable in 1 second, but the
> results may not come back for between 1 and 10 seconds" is much more
> frustrating.
>
> FWIW,
> Erick
>
> On Sat, May 6, 2017 at 5:12 AM, Chris Troullis <cptroullis@gmail.com>
> wrote:
> > Hi,
> >
> > I use Solr to serve multiple tenants and currently all tenant's data
> > resides in one large collection, and queries have a tenant identifier.
> This
> > works fine with aggressive autowarming, but I have a need to reduce my
> NRT
> > search capabilities to seconds as opposed to the minutes it is at now,
> > which will mean drastically reducing if not eliminating my autowarming.
> As
> > such I am considering splitting my index out by tenant so that when one
> > tenant modifies their data it doesn't blow away all of the searcher based
> > caches for all tenants on soft commit.
> >
> > I have done a lot of research on the subject and it seems like Solr Cloud
> > can have problems handling large numbers of collections. I'm obviously
> > going to have to run some tests to see how it performs, but my main
> > question is this: are there pros and cons to splitting the index into
> > multiple collections vs having 1 collection but splitting into multiple
> > shards? In my case I would have a shard per tenant and use implicit
> routing
> > to route to that specific shard. As I understand it a shard is basically
> > it's own lucene index, so I would still be eating that overhead with
> either
> > approach. What I don't know is if there are any other overheads involved
> > WRT collections vs shards, routing, zookeeper, etc.
> >
> > Thanks,
> >
> > Chris
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message