lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shawn Heisey <s...@elyograg.org>
Subject Re: Indexing 4 different cores same machine
Date Thu, 09 May 2013 01:38:52 GMT
On 5/8/2013 8:12 AM, marotosg wrote:
> Hi,
> 
> I have 4 different cores in same machine. 
> Person core -> 3 million docs   -> 20 GB size
> Company Core  -> 1 million docs -> 2GB size
> Documents Core -> 5 million docs -> 5GB size
> Emails Core -> 50,000 thousand  -> 200 Mb
> 
> While I am indexing data performance in server is almost the same if I am
> indexing only one core or all
> cores at the same time.
> 
> I thought having different cores allow you to get different threads in
> parallel gaining some performance.
> Am I right?. My server is never reaching 100% CPU use. It always about 50%
> or even less.
> I had a look to I/O and it is not a problem.

You say that I/O performance appears to be good, but I/O is still likely
the bottleneck here.  When you are indexing them sequentially, each one
has access to full I/O resources, so each one goes at top speed.  If you
do them all at the same time, then they are competing for I/O resources,
so one can do its thing and the others have to wait until the I/O
scheduler can work on their requests.

In most cases, Solr is I/O bound, and the fact that it takes the same
amount of time either way is additional support for the idea that you
are limited by I/O resources, not CPU resources.  Your I/O system is
keeping up, which is good.  If it weren't keeping up, parallel indexing
would actually take even longer.

Thanks,
Shawn


Mime
View raw message