lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jack Krupansky <jack.krupan...@gmail.com>
Subject Re: Limitation on Collections Number
Date Sun, 14 Jun 2015 14:21:10 GMT
As a general rule, there are only two ways that Solr scales to large
numbers: large number of documents and moderate number of nodes (shards and
replicas). All other parameters should be kept relatively small, like
dozens or low hundreds. Even shards and replicas should probably kept down
to that same guidance of dozens or low hundreds.

Tens of millions of documents should be no problem. I recommend 100 million
as the rough limit of documents per node. Of course it all depends on your
particular data model and data and hardware and network, so that number
could be smaller or larger.

The main guidance has always been to simply do a proof of concept
implementation to test for your particular data model and data values.

-- Jack Krupansky

On Sun, Jun 14, 2015 at 7:31 AM, Arnon Yogev <ARNONY@il.ibm.com> wrote:

> We're running some tests on Solr and would like to have a deeper
> understanding of its limitations.
>
> Specifically, We have tens of millions of documents (say 50M) and are
> comparing several "#collections X #docs_per_collection" configurations.
> For example, we could have a single collection with 50M docs or 5000
> collections with 10K docs each.
> When trying to create the 5000 collections, we start getting frequent
> errors after 1000-1500 collections have been created. Feels like some
> limit has been reached.
> These tests are done on a single node + an additional node for replica.
>
> Can someone elaborate on what could limit Solr to a high number of
> collections (if at all)?
> i.e. if we wanted to have 5K or 10K (or 100K) collections, is there
> anything in Solr that can prevent it? Where would it break?
>
> Thanks,
> Arnon

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message