zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joey Echeverria <joe...@gmail.com>
Subject Re: Recommended session timeout
Date Thu, 26 Feb 2009 22:41:03 GMT
I doubt we're swapping, The machine we used to test has 12 GB of RAM
and little else active at the time. I think the main problem is the
heap being too small, so the GC has to run for longer trying to track
down objects to collect.

On Thu, Feb 26, 2009 at 10:28 PM, Benjamin Reed <breed@yahoo-inc.com> wrote:
> just a quick sanity check. are you sure your memory is not overcommitted? in other words
you aren't swapping. since the gc does a bunch of random memory accesses if you swap at all
things will go very slow.
> ben
> ________________________________________
> From: Joey Echeverria [joey42@gmail.com]
> Sent: Thursday, February 26, 2009 1:31 PM
> To: zookeeper-user@hadoop.apache.org
> Subject: Re: Recommended session timeout
> I've answered the questions you asked previously below, but I thought
> I would open with the actual culprit now that we found it. When I said
> loading data before, what I was talking about was sending data via
> Thrift to the machine that was getting disconnected from zookeeper.
> This turned out to be the problem. Too much data was being sent in
> short span of time and this caused memory pressure on the heap. This
> increased the fraction of the time that the GC had to run to keep up.
> During a 143 second test, the GC was running for 33 seconds.
> We found this by running tcpdump on both the machine running the
> ensemble server and the machine connecting to zookeeper as a client.
> We deduced it wasn't a network (lost packet) issue, as we never saw
> unmatched packets in our tests. What did see were "long" 2-7 second
> pauses with no packets being sent. We first attempted to up the
> priority of the zookeeper threads to see if that would help. When it
> didn't, we started monitoring the GC time. We don't have a work around
> yet, other than sending data in smaller batches and  using a longer
> sessionTimeout.
> Thanks for all your help!
> -Joey
>> As an experiment try increasing the timeout to say 30 seconds and re-run
>> your tests. Any change?
> 30 seconds and higher works fine.
>> "loading data" - could you explain a bit more about what you mean by this?
>> If you are able to provide enough information for us to replicate we could
>> try it out (also provide info on your ensemble configuration as Mahadev
>> suggested)
> The ensemble config file looks as follows:
> tickTime=2000
> dataDir=/data/zk
> clientPort=2181
> initLimit=5
> syncLimit=2
> skipACL=true
> server.1=<server>1:2888:3888
> ...
> server.7=<server>7:2888:3888
>> You are referring to startConnect in SendThread?
>> We randomly sleep up to 1 second to ensure that the clients don't all storm
>> the server(s) after a bounce.
> That makes some sense, but it might be worth tweaking that parameter
> based on sessionTimeout since 1 second can easily be 10-20% of
> sessionTimeout.
>> 1) configure your test client to connect to 1 server in the ensemble
>> 2) run the srst command on that server
>> 3) run your client test
>> 4) run the stat command on that server
>> 5) if the test takes some time, run the stat a few times during the test
>>  to get more data points
> The problem doesn't appear to be on the server end as max latency
> never went above 5ms. Also, no messages are shown as queued.

View raw message