lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shawn Heisey <apa...@elyograg.org>
Subject Re: How to Fast Bulk Inserting documents
Date Wed, 19 Aug 2015 18:04:33 GMT
On 8/19/2015 11:09 AM, Troy Edwards wrote:
> I have a requirement where I have to bulk insert a lot of documents in
> SolrCloud.
>
> My average document size is 400 bytes
> Number of documents that need to be inserted 250000/second (for a total of
> about 3.6 Billion documents)
>
> Any ideas/suggestions on how that can be done? (use a client or uploadcsv
> or stream or data import handler)
>
> How can SolrCloud be configured to allow this fast bulk insert?
>
> Any thoughts on what the SolrCloud configuration would probably look like?

I think this is an unrealistic goal, unless you're planning on a couple
hundred shards with a very small number of shards (1 or 2) per server. 
This would also require a very large number of very fast servers with a
fair amount of RAM.  The more shards you have on each server, the more
likely it is that you'll need SSD storage.  This will get very expensive.

It is likely going to take a lot longer than 4 hours to rebuild your
entire 3.6 billion document index.  Your small document size will help
keep the rebuild time lower than I would otherwise expect, but 3.6
billion is a VERY large number.  I can achieve about 6000 docs per
second on my largest index, which means that each of my cold shards
indexes at about 1000 docs per second.  I'm not sure how large my
documents are, but a few kilobytes is probably about right.  The entire
rebuild takes over 9 hours for a little more than 200 million documents.

The best performance is likely to come from a heavily multi-threaded
SolrJ 5.2.1 or later application using CloudSolrClient, with at least
version 5.2.1 on your servers.  Even if you build the hardware
infrastructure I described above, it won't perform to your expectations
unless you've got someone with considerable Java programming skills.

Thanks,
Shawn


Mime
View raw message