lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joseph Obernberger <j...@lovehorsepower.com>
Subject Re: Lost connection to Zookeeper
Date Fri, 05 Jun 2015 13:58:14 GMT
Thank you Shawn!  Yes - it is now a Solr 5.1.0 cloud on 27 nodes and we 
use the startup scripts.  The current index size is 3.0T - about 115G 
per node - index is stored in HDFS which is spread across those 27 nodes 
and about (a guess) - 256 spindles.  Each node has 26G of HDFS cache 
(MaxDirectMemorySize) allocated to Solr.  Zookeeper storage is on local 
disk.  Solr and HDFS run on the same machines. Each node is connected to 
a switch over 1G Ethernet, but the backplane is 40G.
Do you think the clusterstatus and the zookeeper timeouts are related to 
performance issues talking to HDFS?

The JVM parameters are:
-----------------------------------------
-DSTOP.KEY=solrrocks
-DSTOP.PORT=8100
-Dhost=helios
-Djava.net.preferIPv4Stack=true
-Djetty.port=9100
-DnumShards=27
-Dsolr.clustering.enabled=true
-Dsolr.install.dir=/opt/solr
-Dsolr.lock.type=hdfs
-Dsolr.solr.home=/opt/solr/server/solr
-Duser.timezone=UTC-DzkClientTimeout=15000
-DzkHost=eris.querymasters.com:2181,daphnis.querymasters.com:2181,triton.querymasters.com:2181,oberon.querymasters.com:2181,portia.querymasters.com:2181,puck.querymasters.com:2181/solr5


-XX:+CMSParallelRemarkEnabled
-XX:+CMSScavengeBeforeRemark
-XX:+ParallelRefProcEnabled
-XX:+PrintGCApplicationStoppedTime
-XX:+PrintGCDateStamps
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+PrintHeapAtGC
-XX:+PrintTenuringDistribution
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+UseConcMarkSweepGC
-XX:+UseLargePages
-XX:+UseParNewGC-XX:CMSFullGCsBeforeCompaction=1
-XX:CMSInitiatingOccupancyFraction=50
-XX:CMSMaxAbortablePrecleanTime=6000
-XX:CMSTriggerPermRatio=80
-XX:ConcGCThreads=8
-XX:MaxDirectMemorySize=26g
-XX:MaxTenuringThreshold=8
-XX:NewRatio=3
-XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 9100 /opt/solr/server/logs
-XX:ParallelGCThreads=8
-XX:PretenureSizeThreshold=64m
-XX:SurvivorRatio=4
-XX:TargetSurvivorRatio=90
-Xloggc:/opt/solr/server/logs/solr_gc.log
-Xms8g
-Xmx16g
-Xss256k
-verbose:gc
--------------------

The directoryFactory is configured as follows:

<directoryFactory name="DirectoryFactory"
         class="solr.HdfsDirectoryFactory">
         <bool name="solr.hdfs.blockcache.enabled">true</bool>
         <int name="solr.hdfs.blockcache.slab.count">200</int>
         <bool 
name="solr.hdfs.blockcache.direct.memory.allocation">true</bool>
         <int name="solr.hdfs.blockcache.blocksperbank">16384</int>
         <bool name="solr.hdfs.blockcache.read.enabled">true</bool>
         <bool name="solr.hdfs.blockcache.write.enabled">false</bool>
         <bool name="solr.hdfs.nrtcachingdirectory.enable">true</bool>
         <int name="solr.hdfs.nrtcachingdirectory.maxmergesizemb">64</int>
         <int name="solr.hdfs.nrtcachingdirectory.maxcachedmb">512</int>
         <str name="solr.hdfs.home">hdfs://nameservice1:8020/solr5</str>
         <str name="solr.hdfs.confdir">/etc/hadoop/conf.cloudera.hdfs1</str>
     </directoryFactory>

-Joe

On 6/5/2015 9:34 AM, Shawn Heisey wrote:
> On 6/3/2015 6:39 PM, Joseph Obernberger wrote:
>> Hi All - I've run into a problem where every-once in a while one or more
>> of the shards (27 shard cluster) will loose connection to zookeeper and
>> report "updates are disabled".  In additional to the CLUSTERSTATUS
>> timeout errors, which don't seem to cause any issue, this one certainly
>> does as that shard no longer takes any (you guessed it!) updates!
>> We are using Zookeeper with 7 nodes (7 servers in our quorum).
>> There stack trace is:
> Other messages you have sent talk about Solr 5.x, and one of them
> mentions a 16-node cluster with a 2.9 terabyte index, with the index
> data stored on HDFS.
>
> I'm going to venture a guess that you don't have anywhere near enough
> RAM for proper disk caching, leading to general performance issues,
> which ultimately cause timeouts.  With HDFS, I'm not sure whether OS
> disk cache on the Solr server matters very much, or whether that needs
> to be on the HDFS servers.  I would guess the latter.  Also, if your
> storage networking is gigabit or slower, HDFS may have significantly
> more latency than local storage.  For good network storage speed, you
> want 10gig ethernet or Infiniband.
>
> If it's Solr 5.x and you are using the included startup scripts, then
> long GC pauses are probably not a major issue.  The startup scripts
> include significant GC tuning. If you have deployed in your own
> container, GC tuning might be an issue -- it is definitely required.
>
> Here is where I have written down everything I've learned about Solr
> performance problems, most of which are due to one problem or another
> with memory:
>
> https://wiki.apache.org/solr/SolrPerformanceProblems
>
> Is your zookeeper database on local storage or HDFS?  I would suggest
> keeping that on local storage for optimal performance.
>
> Thanks,
> Shawn
>
>


Mime
View raw message