Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@zookeeper.apache.org
Subject: Re: Zookeeper session expiration
To: user@zookeeper.apache.org
References: <DM2PR0101MB08320473E3C246921475B760C53C0@DM2PR0101MB0832.prod.exchangelabs.com>
From: Shawn Heisey <apache@elyograg.org>
Message-ID: <a3cec0d3-0e72-4d88-a112-8e8019f3e6f0@elyograg.org>
Date: Mon, 4 Dec 2017 12:49:26 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101
 Thunderbird/52.5.0
MIME-Version: 1.0
In-Reply-To: <DM2PR0101MB08320473E3C246921475B760C53C0@DM2PR0101MB0832.prod.exchangelabs.com>
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
archived-at: Mon, 04 Dec 2017 19:49:33 -0000

On 12/4/2017 8:22 AM, Anthony Shaya wrote:
> My question is related to how session expiration works, I noticed on many of the client machines the times across these machines were all off (by anywhere from 1 minute to 20 minutes - which was resolved after discovery - haven't verified this completely yet). Can this directly affect session expiration within the zookeeper cluster?
> 
>    *   I read the following in https://wiki.apache.org/hadoop/ZooKeeper/FAQ , "Expirations happens when the cluster does not hear from the client within the specified session timeout period (i.e. no heartbeat).". So in some case it seems like if the times were wrong across the machines its possible one of the clients could of effectively sent a heart beat in the past (not sure about this tbh) and then the cluster expires the session?

I make these comments without any knowledge of what ZK code actually 
does.  I am a member of this list because I'm a representative of the 
Apache Solr project, which uses the ZK client in order to maintain a 
cluster.

IMHO, any software which makes actual decisions based on the timestamps 
in messages from another system is badly designed.  I would hope that 
the ZK designers know this, and always make any decisions related to 
time using the clock in the local system only.

If ZK's designers did the right thing, then a session timeout would 
indicate that quite literally no heartbeats were received in X seconds, 
as measured by the local clock, and the local clock ONLY ... NOT from 
timestamp information received from another system.

Although such a lack of communication could be caused by any number of 
things, including network hardware failure, one of the most common 
reasons I have seen for problems like this is extreme java garbage 
collection pauses in the client software.

Situations where the heap is a little bit too small can cause a java 
program to basically be doing garbage collection constantly, so it 
doesn't have much time to do anything else, like send heartbeats to ZK 
servers.

Situations where the heap is HUGE and garbage collection is not well 
tuned can lead to pauses of a minute or longer while Java does a massive 
full GC.

>    *   I don't have the zookeeper node log for the above time to see what was going on in zookeeper when the cluster determined the session expired.
> 
>    *   Is there any additional logging I can turn on to troubleshoot zk session expiration issues?

Hopefully your ZK clients also have logging.  Failing that, you could 
turn on GC logging for the software with the ZK client (assuming it's a 
Java client) and find a program or website that can examine the log and 
give you statistics or a graph of GC pauses.

If there is a problem in software using the client and whatever logging 
is available doesn't help you figure out what's wrong, you're generally 
going to need to talk to whoever wrote that software for help 
troubleshooting it.

Thanks,
Shawn