lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zheng Lin Edwin Yeo <edwinye...@gmail.com>
Subject Re: Does more shards in core improve performance?
Date Mon, 21 Sep 2015 02:13:38 GMT
I didn't find any increase in indexing throughput by adding shards in the
same machine.

However, I've managed to feed the index to Solr from more than one thread
at a time. It can take up to 3 threads without affecting the indexing
speed. Anything more than that, the CPU will hit 100%, and the indexing
speed in all the threads will be reduced.

Regards,
Edwin


On 18 September 2015 at 19:38, Gili Nachum <gilinachum@gmail.com> wrote:

> If cpu is just 50% and adding a shard does increase indexing throughput
> then check for disk bottleneck.
> On Sep 17, 2015 18:19, "Zheng Lin Edwin Yeo" <edwinyeozl@gmail.com> wrote:
>
> > Thank you everyone for your reply.
> >
> > > How many CPUs on that machine? How many other requests using the
> server?
> >
> > A) There's 8 CPU on the machine, and there is no other requests that's
> > using the server. Only the indexing script is running.
> >
> > > A simple metric is to look at CPU usage on the machine: If it is near
> > 100% when you index, you will need extra hardware to get more speed.
> > If it is substantially less than 100%, then feed Solr from more than one
> > thread at a time.
> >
> > A) So far from what I observe, the CPU usage is usually around 50% to
> 70%.
> > It haven't go up to 100% yet. But I'll probably try to do sharing on a
> > different machine, as that is probably the case for the real production
> > server.
> >
> >
> > Regards,
> > Edwin
> >
> >
> > On 17 September 2015 at 19:55, Toke Eskildsen <te@statsbiblioteket.dk>
> > wrote:
> >
> > > On Thu, 2015-09-17 at 16:58 +0800, Zheng Lin Edwin Yeo wrote:
> > >
> > > > I was trying with 2 shards and 4 shards but all on the same machine,
> > > > and they have the same performance (no improvement in performance) as
> > > > the one with 1 shard. My machine has a 32GB RAM.
> > >
> > > As you are testing indexing speed, Shalin's post is spot-on: Sharding
> on
> > > the same machine won't help you. I just added my comment on search to
> > > help build a complete picture.
> > >
> > > A simple metric is to look at CPU usage on the machine: If it is near
> > > 100% when you index, you will need extra hardware to get more speed.
> > > If it is substantially less than 100%, then feed Solr from more than
> one
> > > thread at a time.
> > >
> > > - Toke Eskildsen, State and University Library, Denmark
> > >
> > >
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message