zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Henry May <hj...@us.ibm.com>
Subject RE: Latency in asynchronous mode
Date Thu, 23 Oct 2014 15:28:15 GMT
I'm a ZK newbie, but I have a hypothesis to test.  At least on my 
synchronous standalone ZK server I've been able to correlate ZK latency to 
disk write response time.  I have a perl script that harvests the ZK stats 
on regular intervals.  At the beginning of each interval it does srst so I 
truly get the max latency on that interval.  Whenever I find an interval 
with long max latency, I can invariably find an egregiously long disk IO 
operation in a block IO trace.

What does ZK latency measure?  Does it measure the delta between the 
arrival of the request and the completion of the request?  The block IO 
traces are showing a single thread in the ZK process doing all of the 
writes synchronously.  This thread spends about 90% of it's time waiting 
for disk IO.  If the ZK latency timer starts ticking as soon as the 
request arrives from the client, then that incoming request has to wait 
for all outstanding requests ahead of it to complete, and that time is 
accumulated in latency.  Effectively, latency includes disk queueing time 
as well as service time.  Does that make any sense? 

(Maybe this is exactly what Alexander is saying, but I had composed this 
note by the time I saw his post and had to chime in.)

Henry May
IBM InfoSphere Streams Performance
hjmay@us.ibm.com
720-342-8873
Tie: 963-8873




From:   Alexander Shraer <shralex@gmail.com>
To:     user@zookeeper.apache.org
Date:   10/23/2014 10:12 AM
Subject:        RE: Latency in asynchronous mode



I still stay with my previous explanation :) in async mode each client
invokes many ops concurrently resulting in a longer queue at the leader
On Oct 23, 2014 3:32 PM, "Ibrahim El-sanosi (PGR)" <
i.s.el-sanosi@newcastle.ac.uk> wrote:

> Thank you Alexander for  replay,
>
> In fact, I use more than one clients (one, two, three, four ......., 
ten),
> in both modes (synchronous  and asynchronous). So, I found the latency 
in
> Asynchronous Mode is much higher than latency in synchronous   mode. I 
am
> really wondering why I am getting such a big different.
>
> In synchronous  mode, the latency vary between min/avr/max=5/20/50 and
> min/avr/max=11/50/120, but it is never reach min/avg/max: 1/371/627 as 
in
> asynchronous  mode.
>
> Any thought?
>
> Thank you
>
> -----Original Message-----
> From: Alexander Shraer [mailto:shralex@gmail.com]
> Sent: Thursday, October 23, 2014 02:14 م
> To: user@zookeeper.apache.org
> Subject: Re: Latency in asynchronous mode
>
> Maybe due to queueing at the leader in asynchronous mode - if in your
> experiment you have one client in sync mode the leader has just one op 
in
> the queue at a time On Oct 23, 2014 1:57 PM, "Ibrahim" <
> i.s.el-sanosi@newcastle.ac.uk> wrote:
>
> > Hi folks,
> >
> > I am testing ZooKeeper latency in Asynchronous mode. I am sending
> > update
> > (write) requests to Zookeeper cluster that consists of 5 physical
> > Zookeeper.
> >
> > So, when I run the stat command I get high latency like:
> > Latency min/avg/max: 7/339/392
> > Latency min/avg/max: 1/371/627
> > Latency min/avg/max: 1/371/627
> > Latency min/avg/max: 1/364/674
> > I guess such high latency correspond to fsync (batch requests). But I
> > wish if someone could help me and explain this behaviour.
> >
> > However, testing Zookeeper using Synchronous mode, it gives me
> > reasonable result like:
> > Latency min/avg/max: 6/24/55
> > Latency min/avg/max: 7/22/61
> > Latency min/avg/max: 7/30/65
> >
> > Note that the latency measures in milliseconds.
> >
> > I look forward to hearing from you.
> >
> > Ibrahim
> >
> >
> >
> >
> >
> >
> >
> > --
> > View this message in context:
> > http://zookeeper-user.578899.n2.nabble.com/Latency-in-asynchronous-mod
> > e-tp7580446.html Sent from the zookeeper-user mailing list archive at
> > Nabble.com.
> >
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message