lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Toke Eskildsen ...@statsbiblioteket.dk>
Subject Re: Whether solr can support 2 TB data?
Date Sat, 24 Sep 2016 14:23:23 GMT
Jeffery Yuan <yuanyun.cn@gmail.com> wrote:
>  In our application, every data there is about 800mb raw data, we are going
> to store this data for 5 years, then it's about 1 or 2 TB data.

>  I am wondering whether solr can support this much data?

Yes it can.

Or rather: You could probably construct a scenario where it is not feasible, but you would
have to be very creative.

>  Usually how much data we store per node, how many nodes we can have in
> solr cloud, what hardware configuration each node should be?

As Shawn states, it is very hard to give advice on hardware (and I applaud him from refraining
from giving the usual "free RAM == index size"-advice). However, we love to guesstimate, but
to do that you really need to provide more details.


2TB of index that has hundreds of concurrent users, thousands of updates per seconds and heavy
aggregations (grouping, faceting, streaming...) is a task that takes experimentation and beefy
hardware.

2TB of index that is rarely updated and accessed by a few people at a time, which are okay
with multi-second response times, can be handled by a desktop-class machine with SSDs.


Tell us about query types, update rates, latency requirements, document types and concurrent
users. Then we can begin to guess.

- Toke Eskildsen

Mime
View raw message