zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akmal Abbasov <akmal.abba...@icloud.com>
Subject Re: Transaction timeouts
Date Tue, 17 Nov 2015 20:13:25 GMT
Hi Raul,
Thank you for your response.
I am running zookeeper with -Xms512m -Xmx1g options, is this enough.
Regarding the network, all of the server zk server nodes are hosted in the cloud, in the same
dc.
But according to the zk troubleshooting guide, the timeout should be increased for cloud environments.
One more thing is that, I’m seeing a lot of 
fsync-ing the write ahead log in SyncThread:1 took 2962ms which will adversely effect operation
latency. See the ZooKeeper troubleshooting guide
messages in the logs.
Thank you.

Regards, Akmal

> On 17 Nov 2015, at 19:10, Raúl Gutiérrez Segalés <rgs@itevenworks.net> wrote:
> 
> Hi,
> 
> On 17 November 2015 at 05:10, Akmal Abbasov <akmal.abbasov@icloud.com>
> wrote:
> 
>> Hi, I’m seeing a lot of `Closing connection to peer due to transaction
>> timeout` messages in zk logs, in all zk servers.
>> Is this transaction timeout configured through syncLimit in zk config file.
>> 
> 
> That message comes from LearnerHandler#ping() [0], and the frequency of
> pings from the leader to learners
> is twice a tick [1]. So if your tickTime is 2000ms (the default), you are
> pinging the learners every second. You could
> adjust the tickTime and see if it gets better. But I suspect something else
> (GC-ing? noisy network?) is going on, given that it
> shouldn't be that hard for the leader and learners to keep up with 1 ping
> every sec.
> 
> You can check ZAB messages (i.e.: pings, acks, commits, proposals, etc.)
> between the leader and learners using zktraffic's
> zk-dump [2].
> 
> 
>> Also does zk server need to be restarted in order to update this config?
>> 
> 
> yes.
> 
> 
> -rgs
> 
> [0]
> https://github.com/apache/zookeeper/blob/trunk/src/java/main/org/apache/zookeeper/server/quorum/LearnerHandler.java#L923
> [1]
> https://github.com/apache/zookeeper/blob/trunk/src/java/main/org/apache/zookeeper/server/quorum/Leader.java#L549
> [2] https://github.com/twitter/zktraffic


Mime
View raw message