lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Ingestion not scaling horizontally as I add more cores to Solr
Date Wed, 10 Jan 2018 21:43:31 GMT
And I'd add
- are you sending one document at a time or batching them up? See:
https://lucidworks.com/2015/10/05/really-batch-updates-solr-2/

Best,
Erick

On Wed, Jan 10, 2018 at 1:35 PM, Gus Heck <gus.heck@gmail.com> wrote:

> Ok then here's a few things to check...
>
>    - Did you sept up an actual multiple node cluster or are you running
>    this all on one box?
>    - Are you configuring Jmeter to send with multiple threads?
>    - Are they all sending to the same node, or are you distributing across
>    nodes? Is there a load balancer?
>    - Are you sending from a machine on the same network as the machines in
>    the Solr cluster?
>    - If you are sending requests up to the cloud from your local machine,
>    that is frequently a slow link.
>    - Also don't forget to check your zookeeper cluster's health... if it's
>    bogged down that will slow down solr.
>
> If you have all machines on the same network, many threads, load balancing
> and no questionable equipment (or networking limitations put in place by
> IT) in the middle, then something (either CPU or network interface) should
> be maxed out somewhere on at least one machine, either on the Jmeter side
> or Solr side.
>
> -Gus
>
> On Wed, Jan 10, 2018 at 3:54 PM, Shashank Pedamallu <spedamallu@vmware.com
> >
> wrote:
>
> > Hi Gus,
> >
> > Thank  for the reply. I’m sending via jmeter running on my local machine
> > to Solr running on a remote vm.
> >
> > Thanks,
> > Shashank
> >
> > On 1/10/18, 12:34 PM, "Gus Heck" <gus.heck@gmail.com> wrote:
> >
> >     Ingested how? Sounds like your document sending mechanism is maxed,
> > not the
> >     solr cluster...
> >
> >     On Wed, Jan 10, 2018 at 2:58 PM, Shashank Pedamallu <
> > spedamallu@vmware.com>
> >     wrote:
> >
> >     > Hi,
> >     >
> >     >
> >     >
> >     > I’m trying to find the upper thresholds of ingestion and I have
> > tried the
> >     > following. In each of the experiments, I’m ingesting random
> > documents with
> >     > 5 fields.
> >     >
> >     >
> >     > Number of Cores Number of documents ingested per second per core
> >     > 1       89000
> >     > 3       33000
> >     > 5       18000
> >     >
> >     >
> >     > As you can see, the number of documents being ingested per core is
> > not
> >     > scaling horizontally as I'm adding more cores. Rather the total
> > number of
> >     > documents getting ingested for Solr JVM is being topped around 90k
> >     > documents per second.
> >     >
> >     >
> >     > From the iostats and top commands, I do not see any bottlenecks
> with
> > the
> >     > iops or cpu respectively, CPU usaeg is around 65% and a sample of
> > iostats
> >     > is below:
> >     >
> >     > avg-cpu:  %user   %nice %system %iowait  %steal   %idle
> >     >
> >     >           55.32    0.00    2.33    1.64    0.00   40.71
> >     >
> >     >
> >     > Device:            tps    kB_read/s    kB_wrtn/s    kB_read
> > kB_wrtn
> >     >
> >     > sda5           2523.00     45812.00    298312.00      45812
> >  298312
> >     >
> >     >
> >     > Can someone please guide me as to how I can debug this further and
> >     > root-cause the bottleneck for not being able to increase the
> > ingestion
> >     > horizontally.
> >     >
> >     >
> >     > Thanks,
> >     >
> >     > Shashank
> >     >
> >
> >
> >
> >     --
> >     https://urldefense.proofpoint.com/v2/url?u=http-3A__www.
> > the111shift.com&d=DwIFaQ&c=uilaK90D4TOVoH58JNXRgQ&r=
> > blJD2pBapH3dDkoajIf9mT9SSbbs19wRbChNde1ErNI&m=DT_
> > 33Z3k4h8T1t65CuyH0oMxay15ddkfDYAQefzgpa4&s=6-1wd3YPVRgcvlk3LkK7Wz-
> > 3hDFliEGwVGc44HJH1x4&e=
> >
> >
> >
>
>
> --
> http://www.the111shift.com
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message