zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ibrahim El-sanosi (PGR)" <i.s.el-san...@newcastle.ac.uk>
Subject RE: Latency in asynchronous mode
Date Thu, 23 Oct 2014 18:21:00 GMT
Hi Rakesh,

First of all, the zookeeper ensemble consists of five Zookeeper servers. Also I have another
10 clients machines used to send write requests to Zookeeper. The benchmark code creates 5
threads (equal to number of Zookeeper server) , each thread associates with one Zookeeper
server.  So, in this case, each zookeeper server will receive a set of write requests. The
benchmark code runs for 30 seconds. 

 Async tests: 

* Number of clients
In fact, I have different test, each test has different number of clients. For example, the
bellow shows the latency corresponds to different number of clients:
Five clients: Latency min/avg/max: 235/366/515
Ten clients:  Latency min/avg/max: 252/368/505

* Number of threads
As explained above, each client creates 5 threads and each thread connects to one Zookeeper
server. For instance, test using 5 clients’ machines, each Zookeeper server receives five
threads. 

* data size storing in each znode
The data size store in znode is 100 bytes

Also, it would be good to monitor :

1) JVM stats(one way is through JMX) like heap, gc activities. This is to see if latency spike
corresponds to gc activity or not.

If you mean by JVM stats the four word stat command, then  the latency result showed above
is generated using this command. If you mean something else then I have to read about and
tell you late on. 

2) Since you are doubting fsync, I think $ iostat would be helpful to see disk statistics.
For example, $ iostat -d -x 2 10 and collects the disk latency.

Yes, the batch size that I use in SyncrequestProcessor class is 1000 requests. I think this
is preferable size. Also, I will try to use iostat.
 
3) CPU usage through top or sar unix commands. I didn't use sar , but I could see it gives
more details like percent of CPU running idle with a process waiting for block I/O etc.

Yes, I will use the top command to gathering the resource utilization. However, I don’t
think top or sar will answer my question. Because I am thinking there is different between
Asynchroned and Synchronized mode for measuring the latency.  

Thank you for your attention

I look forward to hearing from you


Ibrahim

-----Original Message-----
From: Rakesh Radhakrishnan [mailto:rakeshr.apache@gmail.com] 
Sent: Thursday, October 23, 2014 03:58 م
To: user@zookeeper.apache.org
Subject: Re: Latency in asynchronous mode

Hi Ibrahim,

In async tests, could you give the details like:

* number of clients
* number of threads
* data size storing in each znode

Also, it would be good to monitor :

1) JVM stats(one way is through JMX) like heap, gc activities. This is to see if latency spike
corresponds to gc activity or not.

2) Since you are doubting fsync, I think $ iostat would be helpful to see disk statistics.
For example, $ iostat -d -x 2 10 and collects the disk latency.

3) CPU usage through top or sar unix commands. I didn't use sar , but I could see it gives
more details like percent of CPU running idle with a process waiting for block I/O etc.


-Rakesh


On Thu, Oct 23, 2014 at 6:44 PM, Alexander Shraer <shralex@gmail.com> wrote:

> Maybe due to queueing at the leader in asynchronous mode - if in your 
> experiment you have one client in sync mode the leader has just one op 
> in the queue at a time On Oct 23, 2014 1:57 PM, "Ibrahim" 
> <i.s.el-sanosi@newcastle.ac.uk> wrote:
>
> > Hi folks,
> >
> > I am testing ZooKeeper latency in Asynchronous mode. I am sending 
> > update
> > (write) requests to Zookeeper cluster that consists of 5 physical 
> > Zookeeper.
> >
> > So, when I run the stat command I get high latency like:
> > Latency min/avg/max: 7/339/392
> > Latency min/avg/max: 1/371/627
> > Latency min/avg/max: 1/371/627
> > Latency min/avg/max: 1/364/674
> > I guess such high latency correspond to fsync (batch requests). But 
> > I
> wish
> > if someone could help me and explain this behaviour.
> >
> > However, testing Zookeeper using Synchronous mode, it gives me 
> > reasonable result like:
> > Latency min/avg/max: 6/24/55
> > Latency min/avg/max: 7/22/61
> > Latency min/avg/max: 7/30/65
> >
> > Note that the latency measures in milliseconds.
> >
> > I look forward to hearing from you.
> >
> > Ibrahim
> >
> >
> >
> >
> >
> >
> >
> > --
> > View this message in context:
> >
> http://zookeeper-user.578899.n2.nabble.com/Latency-in-asynchronous-mod
> e-tp7580446.html
> > Sent from the zookeeper-user mailing list archive at Nabble.com.
> >
>
Mime
View raw message