lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Indexing 700 docs per second
Date Tue, 19 Apr 2016 17:12:46 GMT
Make very sure you batch updates though.
Here's a benchmark I ran:
https://lucidworks.com/blog/2015/10/05/really-batch-updates-solr-2/

NOTE: it's not entirely clear that you want to
put 122M docs on a single shard. Depending on the queries
you'll run you may want 2 or more shards, but that depends
on the query pattern and your SLAs. Here's the long version
of "you really have to load test this":
https://lucidworks.com/blog/2012/07/23/sizing-hardware-in-the-abstract-why-we-dont-have-a-definitive-answer/

Best,
Erick

On Tue, Apr 19, 2016 at 6:48 AM, Susheel Kumar <susheel2777@gmail.com> wrote:
>  It sounds achievable with your machine configuration and i would suggest
> to try out atomic update.  Use SolrJ with multi-threaded indexing for
> higher indexing rate.
>
> Thanks,
> Susheel
>
>
>
> On Tue, Apr 19, 2016 at 9:27 AM, Tom Evans <tevans.uk@googlemail.com> wrote:
>
>> On Tue, Apr 19, 2016 at 10:25 AM, Mark Robinson <mark123learns@gmail.com>
>> wrote:
>> > Hi,
>> >
>> > I have a requirement to index (mainly updation) 700 docs per second.
>> > Suppose I have a 128GB RAM, 32 CPU machine, with each doc size around 260
>> > byes (6 fields out of which only 2 will undergo updation at the above
>> > rate). This collection has around 122Million docs and that count is
>> pretty
>> > much a constant.
>> >
>> > 1. Can I manage this updation rate with a non-sharded ie single Solr
>> > instance set up?
>> > 2. Also is atomic update or a full update (the whole doc) of the changed
>> > records the better approach in this case.
>> >
>> > Could some one please share their views/ experience?
>>
>> Try it and see - everyone's data/schemas are different and can affect
>> indexing speed. It certainly sounds achievable enough - presumably you
>> can at least produce the documents at that rate?
>>
>> Cheers
>>
>> Tom
>>

Mime
View raw message