Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@zookeeper.apache.org
From: Akmal Abbasov <akmal.abbasov@icloud.com>
Content-type: multipart/alternative;
 boundary="Apple-Mail=_1C36DD9E-70F9-43D0-963A-24223C14E0AA"
Message-id: <04FC096E-EC96-4694-BCD9-8DA0609F03B2@icloud.com>
MIME-version: 1.0 (Mac OS X Mail 8.2 \(2104\))
Subject: Re: Transaction timeouts
Date: Wed, 18 Nov 2015 11:37:01 +0100
References: <CCE4223E-FC00-4CB5-BD7A-F6D0595AC4AF@icloud.com>
 <CAMUR=Wi=JXdfdUt0kBHKFcCMKgT4X61wKkRp8riQTky8MRd4HA@mail.gmail.com>
 <39D690DF-62C9-4923-8AC2-7EE609A0BEC4@icloud.com>
 <CAMUR=WicBrMQwGBBwsV-N=oGRO9Zdn3RtqwPnr36zDrS1MXo-g@mail.gmail.com>
To: user@zookeeper.apache.org
In-reply-to: 
 <CAMUR=WicBrMQwGBBwsV-N=oGRO9Zdn3RtqwPnr36zDrS1MXo-g@mail.gmail.com>

--Apple-Mail=_1C36DD9E-70F9-43D0-963A-24223C14E0AA
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8


> On 17 Nov 2015, at 21:34, Ra=C3=BAl Guti=C3=A9rrez Segal=C3=A9s =
<rgs@itevenworks.net> wrote:
>=20
> On 17 November 2015 at 12:13, Akmal Abbasov <akmal.abbasov@icloud.com>
> wrote:
>=20
>> Hi Raul,
>> Thank you for your response.
>> I am running zookeeper with -Xms512m -Xmx1g options, is this enough.
>>=20
>=20
> It depends on your workload.. how many writes/read per sec are you
> expecting/seeing? Are you seeing long
> GC pauses? If so, you'll need more mem or bigger tick times, otherwise
> you'll miss the deadlines for the
> pings (both among learners and to clients=E2=80=A6)
>=20
Where I can find this information, in fact information regarding =
read/writes.=20
This is the output of the stat command
Server 1
Latency min/avg/max: 0/66/5212
Received: 8722
Sent: 8694
Connections: 19
Outstanding: 0
Zxid: 0xa9600002ef2
Mode: follower
Node count: 479

Server 2=20
Latency min/avg/max: 0/70/5252
Received: 8228
Sent: 8203
Connections: 16
Outstanding: 0
Zxid: 0xa9600002e12
Mode: leader
Node count: 479

Server 3
Latency min/avg/max: 0/0/1
Received: 140
Sent: 139
Connections: 2
Outstanding: 0
Zxid: 0xa9600002bf8
Mode: follower
Node count: 479

All the servers have the same configs.=20
Is -Xms512m -Xmx1g enough to handle my workload.
Moreover I see that the load is not evenly distributed. Is it something =
that should be tuned manually,
or there is something like hbase/hdfs balancer, which will take care of =
this?

>=20
>> Regarding the network, all of the server zk server nodes are hosted =
in the
>> cloud, in the same dc.
>> But according to the zk troubleshooting guide, the timeout should be
>> increased for cloud environments.
>>=20
>=20
> Yup, latency can be unpredictable in the cloud=E2=80=A6
>=20
>=20
>> One more thing is that, I=E2=80=99m seeing a lot of
>> fsync-ing the write ahead log in SyncThread:1 took 2962ms which will
>> adversely effect operation latency. See the ZooKeeper troubleshooting =
guide
>> messages in the logs.
>>=20
>=20
> That definitely looks bad and will block everything else. What type of =
disc
> are you writing your logs and snapshots to? Are they
> separate volumes?
I=E2=80=99m using separate disk for both logs and data. But they=E2=80=99r=
e hdd, not ssd.=20
So my assumption=20

I=E2=80=99ve tried to understand what actually is happening, here is the =
summary of the logs
08:22:08,201	Transaction timeout
08:22:08,596 - 08:22:25,441	ZookeeperServer not running
08:22:24,927	New election
Everything is starting from =E2=80=99Transaction timeout=E2=80=99 in =
leader, which caused =E2=80=98Exception when following the leader=E2=80=99=
 in learners.
Then all zookeeper processes are shutting down. New election is =
happening and zookeeper processes are starting.=20

And one more thing, what=E2=80=99s the best way to update the configs =
without downtime.
Thank you.

Regards, Akmal

=09


--Apple-Mail=_1C36DD9E-70F9-43D0-963A-24223C14E0AA--