lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "bidorbuy (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-11078) Solr query performance degradation since Solr 6.4.2
Date Thu, 02 Nov 2017 05:53:02 GMT

    [ https://issues.apache.org/jira/browse/SOLR-11078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16235224#comment-16235224
] 

bidorbuy commented on SOLR-11078:
---------------------------------

We have 3 identical production servers. Two are running Solr 6.4.2 where the schema is still
using Trie* fields. The 3rd server where we have the performance degradation running any version
of Solr since 6.4.2 has now been upgraded to Solr 7.1.0 where Trie*-fields have been changed
to *Point-fields. Index has been rebuild and Solr performance is worse compared to Solr 6.4.2.

All servers are running:
- 10 GB RAM, 4 CPUs
- CentOS 7.3.1611
- Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
- Solr Heap -Xms/Xmx is set to 4GB. GC tuning as per attached solr.in.sh (the settings are
the same between 6.4.2 and 7.1.0)
- Two Solr indices: tradesearch (5.5m documents and 3,4GB size) and searchsuggestions (2.7m
documents and 700MB size)

Switching Solr 7.1.0 into production load:
* Load average (via top) shot to 19 and after a few minutes "settled" at 12 (on 6.4.2 it is
about 2.2 - 4)
* Average query time is about 230ms - this is 4 times slower compared to 6.4.2 (on 6.4.2 it
is about 50ms)
* A specific query (we know it is generally slow) runs between 8-10 seconds (on 6.4.2 it would
that about 1.3 seconds)

I can switch 7.1.0 into parallel production tests to collate more data. Any help is appreciated.

> Solr query performance degradation since Solr 6.4.2
> ---------------------------------------------------
>
>                 Key: SOLR-11078
>                 URL: https://issues.apache.org/jira/browse/SOLR-11078
>             Project: Solr
>          Issue Type: Bug
>      Security Level: Public(Default Security Level. Issues are Public) 
>    Affects Versions: 6.6
>         Environment: * CentOS 7.3 (Linux zasolrm03 3.10.0-514.26.2.el7.x86_64 #1 SMP
Tue Jul 4 15:04:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux)
> * Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
> * 4 CPU, 10GB RAM
> Running Solr 6.6.0 with the following JVM settings:
> java -server -Xms4G -Xmx4G -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90
-XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4
-XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled
-XX:+ParallelRefProcEnabled -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps
-XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/home/prodza/solrserver/../logs/solr_gc.log
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=9 -XX:GCLogFileSize=20M -Dsolr.log.dir=/home/prodza/solrserver/../logs
-Djetty.port=8983 -DSTOP.PORT=7983 -DSTOP.KEY=solrrocks -Duser.timezone=SAST -Djetty.home=/home/prodza/solrserver/server
-Dsolr.solr.home=/home/prodza/solrserver/../solr -Dsolr.install.dir=/home/prodza/solrserver
-Dlog4j.configuration=file:/home/prodza/solrserver/../config/log4j.properties -Xss256k -Xss256k
-Dsolr.log.muteconsole -XX:OnOutOfMemoryError=/home/prodza/solrserver/bin/oom_solr.sh 8983
/home/prodza/solrserver/../logs -jar start.jar --module=http
>            Reporter: bidorbuy
>            Priority: Major
>         Attachments: compare-6.4.2-6.6.0.png, core-admin-tradesearch.png, jvm-stats.png,
schema.xml, screenshot-1.png, screenshot-2.png, solr-sample-warning-log.txt, solrconfig.xml
>
>
> We are currently running 2 separate Solr servers - refer to screenshots:
> * zasolrm02 is running on Solr 6.4.2
> * zasolrm03 is running on Solr 6.6.0
> Both servers have the same OS / JVM configuration and are using their own indexes. We
round-robin load-balance through our Tomcats and notice that Since Solr 6.4.2 performance
has dropped. We have two indices per server "searchsuggestions" and "tradesearch". There is
a noticeable drop in performance since Solr 6.4.2.
> I am not sure if this is perhaps related to metric collation or other underlying changes.
I am not sure if other high transaction users have noticed similar issues.
> *1) zasolrm03 (6.6.0) is almost twice as slow on the tradesearch index:*
> !compare-6.4.2-6.6.0.png!
> *2) This is also visible in the searchsuggestion index:*
> !screenshot-1.png!
> *3) The Tradesearch index shows the biggest difference:*
> !screenshot-2.png!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message