zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Hunt <ph...@apache.org>
Subject Re: Recommended session timeout
Date Thu, 26 Feb 2009 21:47:46 GMT
That's very interesting results, a good job sleuthing. You might try the 
concurrent collector?
http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning_6.html#available_collectors.selecting

specifically item 4  "-XX:+UseConcMarkSweepGC"

I've never used this before myself but it's supposed to reduce the gc 
pauses to less than a second. Might require some tuning though...

Patrick


Joey Echeverria wrote:
> I've answered the questions you asked previously below, but I thought
> I would open with the actual culprit now that we found it. When I said
> loading data before, what I was talking about was sending data via
> Thrift to the machine that was getting disconnected from zookeeper.
> This turned out to be the problem. Too much data was being sent in
> short span of time and this caused memory pressure on the heap. This
> increased the fraction of the time that the GC had to run to keep up.
> During a 143 second test, the GC was running for 33 seconds.
> 
> We found this by running tcpdump on both the machine running the
> ensemble server and the machine connecting to zookeeper as a client.
> We deduced it wasn't a network (lost packet) issue, as we never saw
> unmatched packets in our tests. What did see were "long" 2-7 second
> pauses with no packets being sent. We first attempted to up the
> priority of the zookeeper threads to see if that would help. When it
> didn't, we started monitoring the GC time. We don't have a work around
> yet, other than sending data in smaller batches and  using a longer
> sessionTimeout.
> 
> Thanks for all your help!
> 
> -Joey
> 
>> As an experiment try increasing the timeout to say 30 seconds and re-run
>> your tests. Any change?
> 
> 30 seconds and higher works fine.
> 
>> "loading data" - could you explain a bit more about what you mean by this?
>> If you are able to provide enough information for us to replicate we could
>> try it out (also provide info on your ensemble configuration as Mahadev
>> suggested)
> 
> The ensemble config file looks as follows:
> 
> tickTime=2000
> dataDir=/data/zk
> clientPort=2181
> initLimit=5
> syncLimit=2
> skipACL=true
> 
> server.1=<server>1:2888:3888
> ...
> server.7=<server>7:2888:3888
> 
>> You are referring to startConnect in SendThread?
>>
>> We randomly sleep up to 1 second to ensure that the clients don't all storm
>> the server(s) after a bounce.
> 
> That makes some sense, but it might be worth tweaking that parameter
> based on sessionTimeout since 1 second can easily be 10-20% of
> sessionTimeout.
> 
>> 1) configure your test client to connect to 1 server in the ensemble
>> 2) run the srst command on that server
>> 3) run your client test
>> 4) run the stat command on that server
>> 5) if the test takes some time, run the stat a few times during the test
>>  to get more data points
> 
> The problem doesn't appear to be on the server end as max latency
> never went above 5ms. Also, no messages are shown as queued.

Mime
View raw message