lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Toke Eskildsen ...@statsbiblioteket.dk>
Subject Re: Whether SolrCloud can support 2 TB data?
Date Sat, 24 Sep 2016 17:48:50 GMT
Regarding a 12TB index:

Yago Riveiro <yago.riveiro@gmail.com> wrote:

> Our cluster is small for the data we hold (12 machines with SSD and 32G of
> RAM), but we don't need sub-second queries, we need facet with high
> cardinality (in worst case scenarios we aggregate 5M unique string values)

> In a peak of inserts we can handle around 25K docs per second without any
> issue with 2 replicas and without compromise reads or put a node in stress.
> Nodes in stress can eject him selfs from the Zookepeer cluster due a GC or a
> lack of CPU to communicate.

I am surprised that you manage to have this working on that hardware. As you have replicas,
it seems to me that you handle 2*12TB of index with 12*32GB of RAM? This is very close to
our setup (22TB of index with 320GB of RAM (updated last week from 256GB) per machine), but
we benefit hugely from having a static index.

I assume the SSDs are local? How much memory do you use for heap on each machine?

- Toke Eskildsen

Mime
View raw message