hadoop-zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Hunt <ph...@apache.org>
Subject Re: zookeeper on ec2
Date Wed, 02 Sep 2009 19:37:34 GMT
How large/small are the writes?

Can you run the following, then try your test again and report the 
results for the time period where your test is running?

iostat -x -d 1

also note that ZK JMX allows you to reset the latency attributes (look 
under "operations" in jconsole). If you reset the latency during your 
test what do you see happen wrt the min/max/avg latency? Keep an eye on 
this (and perhaps resetting the stats every so often) during your test. 
Anything interesting happening that you notice?

I suspect that given a single disk is being used (not a dedicated disk 
for the transaction log), and also given that this host is highly 
virtualized (ec2), it seems to me that the most likely cause is IO. 
Specifically when the zk cluster writes data to disk (due to client 
write) it must sync the transaction log to disk. This sync behavior can 
impact the latency seen by the clients. What type of ec2 node are you 
using? Ted, do you have any insight on this? Any guidelines for the type 
of ec2 node to use for running a zk cluster?


Satish Bhatti wrote:
> According to the jconsole, max memory usage is 30MB, 14 live threads and
> peak CPU usage about 4%, average is under 1%.  We are not really hammering
> it.  Doing about 10 read/writes every second max.
> On Tue, Sep 1, 2009 at 5:20 PM, Ted Dunning <ted.dunning@gmail.com> wrote:
>> This is outrageously large.  Max should be more like 50ms.
>> Either you are doing this somehow, or you have an anomaly on your ZK
>> machine.
>> How much data is in ZK?  How many transaction per seoncd?
>> On Tue, Sep 1, 2009 at 5:11 PM, Satish Bhatti <cthd2001@gmail.com> wrote:
>>> MaxRequestLatency 55767
>> --
>> Ted Dunning, CTO
>> DeepDyve

View raw message