lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Kettmann <andrew.kettm...@evolve24.com>
Subject Re: Solr 7.7.2 Autoscaling policy - Poor performance
Date Tue, 03 Sep 2019 17:10:10 GMT
> How many zookeepers do you have? How many collections? What is there size?
> How much CPU / memory do you give per container? How much heap in comparison to total
memory of the container ?

3 Zookeepers.
733 containers/nodes
735 total cores. Each core ranges from ~4-10GB of index. (Autoscaling splits at 12GB)
10 collections, ranging from 147 shards at most, to 3 at least. Replication factor of 2 other
than .system which has 3 replicas.
Each container has a min/max heap of 750MB other than the overseer containers which have a
min/max of 3GB.
Containers aren't hard limited by K8S on memory or CPU but the machines the containers are
on have 4 cores and ~13GB of ram.

Now that I look at the CPU usage on a per container basis, it looks like it is maxing out
all four cores on the VM that is hosting the overseer container. Barely using the heap (300MB).

I suppose that means that if we put the overseers on machines with more cores, it might be
able to get things done a bit faster. Though that still seems like a limited solution as we
are going to grow this cluster at least double in size if not larger.

We are using the solr:7.7.2 container.

Java Options on the home page are below:
    -DHELM_CHART=overseer
    -DSTOP.KEY=solrrocks
    -DSTOP.PORT=7983
    -Dhost=overseer-solr-0.solr.DOMAIN
    -Djetty.home=/opt/solr/server
    -Djetty.port=8983
    -Dsolr.data.home=
    -Dsolr.default.confdir=/opt/solr/server/solr/configsets/_default/conf
    -Dsolr.install.dir=/opt/solr
    -Dsolr.jetty.https.port=8983
    -Dsolr.log.dir=/opt/solr/server/logs
    -Dsolr.log.level=INFO
    -Dsolr.solr.home=/opt/solr/server/home
    -Duser.timezone=UTC
    -DzkClientTimeout=60000
    -DzkHost=zookeeper-1.DOMAIN:2181,zookeeper-2.DOMAIN:2181,zookeeper-3.DOMAIN:2181/ZNODE
    -XX:+CMSParallelRemarkEnabled
    -XX:+CMSScavengeBeforeRemark
    -XX:+ParallelRefProcEnabled
    -XX:+UseCMSInitiatingOccupancyOnly
    -XX:+UseConcMarkSweepGC
    -XX:-OmitStackTraceInFastThrow
    -XX:CMSInitiatingOccupancyFraction=50
    -XX:CMSMaxAbortablePrecleanTime=6000
    -XX:ConcGCThreads=4
    -XX:MaxTenuringThreshold=8
    -XX:NewRatio=3
    -XX:ParallelGCThreads=4
    -XX:PretenureSizeThreshold=64m
    -XX:SurvivorRatio=4
    -XX:TargetSurvivorRatio=90
    -Xlog:gc*:file=/opt/solr/server/logs/solr_gc.log:time,uptime:filecount=9,filesize=20M
    -Xmx3g
    -Xmx3g
    -Xss256k



evolve24 Confidential & Proprietary Statement: This email and any attachments are confidential
and may contain information that is privileged, confidential or exempt from disclosure under
applicable law. It is intended for the use of the recipients. If you are not the intended
recipient, or believe that you have received this communication in error, please do not read,
print, copy, retransmit, disseminate, or otherwise use the information. Please delete this
email and attachments, without reading, printing, copying, forwarding or saving them, and
notify the Sender immediately by reply email. No confidentiality or privilege is waived or
lost by any transmission in error.

Mime
View raw message