lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: How large is your solr index?
Date Mon, 29 Dec 2014 17:54:00 GMT
When you say 2B docs on a single Solr instance, are you talking only one shard?
Because if you are, you're very close to the absolute upper limit of a
shard, internally
the doc id is an int or 2^31. 2^31 + 1 will cause all sorts of problems.

But yeah, your 100B documents are going to use up a lot of servers...

Best,
Erick

On Mon, Dec 29, 2014 at 7:24 AM, Bram Van Dam <bram.vandam@intix.eu> wrote:
> Hi folks,
>
> I'm trying to get a feel of how large Solr can grow without slowing down too
> much. We're looking into a use-case with up to 100 billion documents
> (SolrCloud), and we're a little afraid that we'll end up requiring 100
> servers to pull it off.
>
> The largest index we currently have is ~2billion documents in a single Solr
> instance. Documents are smallish (5k each) and we have ~50 fields in the
> schema, with an index size of about 2TB. Performance is mostly OK. Cold
> searchers take a while, but most queries are alright after warming up. I
> wish I could provide more statistics, but I only have very limited access to
> the data (...banks...).
>
> I'd very grateful to anyone sharing statistics, especially on the larger end
> of the spectrum -- with or without SolrCloud.
>
> Thanks,
>
>  - Bram

Mime
View raw message