Mailing-List: contact zookeeper-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: zookeeper-user@hadoop.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Message-ID: <49A43906.30406@apache.org>
Date: Tue, 24 Feb 2009 10:14:30 -0800
From: Patrick Hunt <phunt@apache.org>
User-Agent: Thunderbird 2.0.0.19 (X11/20090105)
MIME-Version: 1.0
To: zookeeper-user@hadoop.apache.org
Subject: Re: Recommended session timeout
References: <C5C83638.188CF%mahadev@yahoo-inc.com>
	 <49A32522.7000205@apache.org>
 <92eebe280902232337v2c6e2064oe05775534939cc40@mail.gmail.com>
In-Reply-To: <92eebe280902232337v2c6e2064oe05775534939cc40@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Joey Echeverria wrote:
> Thanks for the link to the documentation. I've been running tests with
> a 5 second session timeout and disconnect events appear frequent. The
> network they're operating on is generally quite, but the disconnects
> to correlate with an increase in activity (e.g. loading data into the
> system).

As an experiment try increasing the timeout to say 30 seconds and re-run 
your tests. Any change?

"loading data" - could you explain a bit more about what you mean by 
this? If you are able to provide enough information for us to replicate 
we could try it out (also provide info on your ensemble configuration as 
Mahadev suggested)

> Does this seem normal to you or does it imply a potential
> configuration problem on my network?

Not enough info at this time to speculate. Can you provide the configs 
for at least 1 server in the ensemble (I'm assuming they are all pretty 
much the same)

> On a related topic, I was reading the 3.1 client source code,
> particularly the reconnect source, and noticed that the client sleeps
> for up to 1 second before trying to reconnect. This seems excessive
> and with a 5 second session timeout leads to more frequent session
> expirations. Almost every time it sleeps for more than about 800 ms, a
> disconnect is followed by an expiration.

You are referring to startConnect in SendThread?

We randomly sleep up to 1 second to ensure that the clients don't all 
storm the server(s) after a bounce.


I suspect that the following is happening:

Your client(s) is sending information to the server, the server has 1 or 
more outstanding requests from the client. You mentioned "loading data", 
at some point the server flushes the data to disk, it could be that this 
flush takes a significant amount of time. As there is communication btw 
the server and client (client sent a request, server is responding) 
there will be no heatbeating server->client going on while the request 
is outstanding (the server will not send a heartbeat because of the 
request being in progress). As a result the client doesn't see a 
response for a potentially long period of time (because of the flush).

Try using the stat & srst commands detailed here:
http://hadoop.apache.org/zookeeper/docs/current/zookeeperAdmin.html#sc_zkCommands

1) configure your test client to connect to 1 server in the ensemble
2) run the srst command on that server
3) run your client test
4) run the stat command on that server
5) if the test takes some time, run the stat a few times during the test
   to get more data points

The stat command will give you min/avg/max latency for requests to the 
server. If max latency goes above your timeout then you will see the 
disconnect on the client. This indicates that the server is probably to 
blame (vs say networking issues, which we see alot). Let us know the 
results of this test.

Btw, latency can be effected by a number of factors:
http://hadoop.apache.org/zookeeper/docs/current/zookeeperAdmin.html#sc_commonProblems

1) make sure the server JDK is not swapping (etc...) as this will kill 
latency
2) is the host(s) running the ZK server on dedicated devices 
(cpu/mem/disk) or sharing resources with other applications?
3) are you using a dedicated transaction log device (drive)? This is 
critical for low-latency & high throughput of the ensemble.

Patrick

> Is this a bug, or desirable behavior?
> 
> Thanks,
> 
> -Joey
> 
> On Mon, Feb 23, 2009 at 10:37 PM, Patrick Hunt <phunt@apache.org> wrote:
>> The latest docs (3.1.0 has some updates to that section) can be found here:
>> http://hadoop.apache.org/zookeeper/docs/r3.1.0/zookeeperProgrammers.html#ch_zkSessions
>>
>> Patrick
>>
>> Mahadev Konar wrote:
>>> Hi Joey,
>>>  here is a link to information on session timeouts.
>>>
>>> http://hadoop.apache.org/zookeeper/docs/r3.0.1/zookeeperProgrammers.html#ch_
>>> zkSessions
>>>  The session timeouts depends on how sensitive you want your application
>>> to
>>> be. A very low session timeout like (1-2 seconds) might lead to your
>>> application being very sensitive to events like minor network problems
>>> etc.,
>>> a higher values of say (30 seconds) on the other hand might lead to slow
>>> detection of client failures -- example one of the zookeeper client which
>>> has ephemeral node goes down, in this case the ephemeral nodes will only
>>> go
>>> away after session timeout.
>>>
>>> I have seen some users using 10-15 seconds of session timeout, but you
>>> should use as per your application requirements.
>>>
>>> Hope this helps.
>>> mahadev
>>>
>>>
>>> On 2/22/09 3:09 AM, "Joey Echeverria" <joey42@gmail.com> wrote:
>>>
>>>> Is there a recommended session timeout? Does it change based on the
>>>> ensemble size?
>>>>
>>>> Thanks,
>>>>
>>>> -Joey