lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Indexing multiple cores simultaneously
Date Sun, 25 Oct 2015 20:45:20 GMT
Let's back up a bit and ask what your primary goal is. Just indexing a
bunch of stuff as fast as possible? By and large, I'd index to a
single core with multiple threads rather than the approach you're
taking (I'm assuming that there's a MERGEINDEXES somewhere in this
process). You should be able to max out your CPUs this way in my
experience.

Have you tried just indexing to a single core with multiple indexing
clients? And what have your results been?

Your issue with 100 cores is probably that you're copying around a
bunch of files as you merge and/or the 100 cores are context switching
to no good purpose.

Best,
Erick

On Sun, Oct 25, 2015 at 12:52 PM, Peri Subrahmanya
<peri.subrahmanya@htcinc.com> wrote:
> Hi,
>
> I wanted to check if the following would work;
>
> 1. Spawn n threads
> 2. Create n-cores
> 3. Index n records simultaneously in n-cores
> 4. Merge all core indexes into a single master core
>
> I have been able to successfully do this for 5 threads (5 cores) with 1000 documents
each. However, I wanted to check if there are any performance parameters that can be tweaked
to make Solr handle 100 cores with 10K records each? It seems to be churning and I am not
sure if it was ever going to finish.
>
> Any ideas on how things might be done differently ?
>
> Thanks
> -Peri
>
> *** DISCLAIMER *** This is a PRIVATE message. If you are not the intended recipient,
please delete without copying and kindly advise us by e-mail of the mistake in delivery.
> NOTE: Regardless of content, this e-mail shall not operate to bind HTC Global Services
to any order or other contract unless pursuant to explicit written agreement or government
initiative expressly permitting the use of e-mail for such purpose.
>
>

Mime
View raw message