zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ibrahim <i.s.el-san...@newcastle.ac.uk>
Subject RE: Latency in asynchronous mode
Date Sat, 25 Oct 2014 11:38:44 GMT
Dear Kishore,

When you said “This operation is expensive can take around 5-20 ms on spinning disk. So
if zookeeper followed a naive way of invoking fsync for every request it would be able to
handle approx 200 transaction per second at max( again depending on the fsync time)”. Here
Are you trying to say at min instead at max. In your scenario, if we multiply 5ms by 200 ops
, we will get 1000 (second), meaning that 200 ops per second.  I may be wrong or missing some
things, can you correct me if so?

I look forward to hearing from you.

Ibrahim

From: kishore g [via zookeeper-user] [mailto:ml-node+s578899n7580453h67@n2.nabble.com]
Sent: Thursday, October 23, 2014 08:16 م
To: Ibrahim El-sanosi (PGR)
Subject: Re: Latency in asynchronous mode

Async api is zookeeper is a way to achieve high through but by trading off
latency. As others have explained, before returning success to the user,
zookeeper always ensures that the entry is flushed into the transaction
log. This operation is expensive can take around 5-20 ms on spinning disk.
So if zookeeper followed a naive way of invoking fsync for every request it
would be able to handle approx 200 transaction per second at max( again
depending on the fsync time). But the server tries to do further
optimization by trying to batch multiple requests in one fsync also
commonly known as group commit. This of course comes impacts the latency
because each request now has to incur some additional latency because the
amount of data written to disk is proportional to the batch size where as
in the sync request each request would write the data proportional to that
request.

So coming back to your question, what you should be really measuring is the
amortized latency on a per write basis from the client. By using sync api,
its unlikely that the group flush is kicking in because the client waits
for the ack of the previous request before sending a new request. So you
are seeing low latency but the total number of writes done during your test
would be low compared to the async api test. In case of async client is not
really waiting for ack for the write in the same thread. So all clients
continue to send requests which means there is a high chance of group flush
to kick in on the server side and because of this the perceived latency can
be higher. For example, if you send 1000 requests back to back in async you
might see the latency as X ms for each request but if you measure the wall
clock time from the start of first request to the last ack got from the
server it would be around X ms.

A good way to understand this is to measure both latency and throughput
(total number of writes) from your client. Also its not clear if the
clients are trying to create new znodes or updating the same one. If they
are updating the same one there might be some conflicts that might create
additional latency. And are the clients doing read and write?. There are
other design choices in zookeeper such a single queue for both reads and
writes that might impact latency as well. It might help if you share the
client code.

I think its important to understand your goal. In general there are two
things one would like to achieve low latency and high throughput. Achieving
both is hard especially when it involves disk io and fsync.

thanks,
Kishore G












On Thu, Oct 23, 2014 at 11:21 AM, Ibrahim El-sanosi (PGR) <
[hidden email]</user/SendEmail.jtp?type=node&node=7580453&i=0>> wrote:

> Hi Rakesh,
>
> First of all, the zookeeper ensemble consists of five Zookeeper servers.
> Also I have another 10 clients machines used to send write requests to
> Zookeeper. The benchmark code creates 5 threads (equal to number of
> Zookeeper server) , each thread associates with one Zookeeper server.  So,
> in this case, each zookeeper server will receive a set of write requests.
> The benchmark code runs for 30 seconds.
>
>  Async tests:
>
> * Number of clients
> In fact, I have different test, each test has different number of clients.
> For example, the bellow shows the latency corresponds to different number
> of clients:
> Five clients: Latency min/avg/max: 235/366/515
> Ten clients:  Latency min/avg/max: 252/368/505
>
> * Number of threads
> As explained above, each client creates 5 threads and each thread connects
> to one Zookeeper server. For instance, test using 5 clients’ machines, each
> Zookeeper server receives five threads.
>
> * data size storing in each znode
> The data size store in znode is 100 bytes
>
> Also, it would be good to monitor :
>
> 1) JVM stats(one way is through JMX) like heap, gc activities. This is to
> see if latency spike corresponds to gc activity or not.
>
> If you mean by JVM stats the four word stat command, then  the latency
> result showed above is generated using this command. If you mean something
> else then I have to read about and tell you late on.
>
> 2) Since you are doubting fsync, I think $ iostat would be helpful to see
> disk statistics. For example, $ iostat -d -x 2 10 and collects the disk
> latency.
>
> Yes, the batch size that I use in SyncrequestProcessor class is 1000
> requests. I think this is preferable size. Also, I will try to use iostat.
>
> 3) CPU usage through top or sar unix commands. I didn't use sar , but I
> could see it gives more details like percent of CPU running idle with a
> process waiting for block I/O etc.
>
> Yes, I will use the top command to gathering the resource utilization.
> However, I don’t think top or sar will answer my question. Because I am
> thinking there is different between Asynchroned and Synchronized mode for
> measuring the latency.
>
> Thank you for your attention
>
> I look forward to hearing from you
>
>
> Ibrahim
>
> -----Original Message-----
> From: Rakesh Radhakrishnan [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=7580453&i=1>]
> Sent: Thursday, October 23, 2014 03:58 م
> To: [hidden email]</user/SendEmail.jtp?type=node&node=7580453&i=2>
> Subject: Re: Latency in asynchronous mode
>
> Hi Ibrahim,
>
> In async tests, could you give the details like:
>
> * number of clients
> * number of threads
> * data size storing in each znode
>
> Also, it would be good to monitor :
>
> 1) JVM stats(one way is through JMX) like heap, gc activities. This is to
> see if latency spike corresponds to gc activity or not.
>
> 2) Since you are doubting fsync, I think $ iostat would be helpful to see
> disk statistics. For example, $ iostat -d -x 2 10 and collects the disk
> latency.
>
> 3) CPU usage through top or sar unix commands. I didn't use sar , but I
> could see it gives more details like percent of CPU running idle with a
> process waiting for block I/O etc.
>
>
> -Rakesh
>
>
> On Thu, Oct 23, 2014 at 6:44 PM, Alexander Shraer <[hidden email]</user/SendEmail.jtp?type=node&node=7580453&i=3>>
> wrote:
>
> > Maybe due to queueing at the leader in asynchronous mode - if in your
> > experiment you have one client in sync mode the leader has just one op
> > in the queue at a time On Oct 23, 2014 1:57 PM, "Ibrahim"
> > <[hidden email]</user/SendEmail.jtp?type=node&node=7580453&i=4>>
wrote:
> >
> > > Hi folks,
> > >
> > > I am testing ZooKeeper latency in Asynchronous mode. I am sending
> > > update
> > > (write) requests to Zookeeper cluster that consists of 5 physical
> > > Zookeeper.
> > >
> > > So, when I run the stat command I get high latency like:
> > > Latency min/avg/max: 7/339/392
> > > Latency min/avg/max: 1/371/627
> > > Latency min/avg/max: 1/371/627
> > > Latency min/avg/max: 1/364/674
> > > I guess such high latency correspond to fsync (batch requests). But
> > > I
> > wish
> > > if someone could help me and explain this behaviour.
> > >
> > > However, testing Zookeeper using Synchronous mode, it gives me
> > > reasonable result like:
> > > Latency min/avg/max: 6/24/55
> > > Latency min/avg/max: 7/22/61
> > > Latency min/avg/max: 7/30/65
> > >
> > > Note that the latency measures in milliseconds.
> > >
> > > I look forward to hearing from you.
> > >
> > > Ibrahim
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > --
> > > View this message in context:
> > >
> > http://zookeeper-user.578899.n2.nabble.com/Latency-in-asynchronous-mod
> > e-tp7580446.html
> > > Sent from the zookeeper-user mailing list archive at Nabble.com.
> > >
> >
>

________________________________
If you reply to this email, your message will be added to the discussion below:
http://zookeeper-user.578899.n2.nabble.com/Latency-in-asynchronous-mode-tp7580446p7580453.html
To unsubscribe from Latency in asynchronous mode, click here<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=7580446&code=aS5zLmVsLXNhbm9zaUBuZXdjYXN0bGUuYWMudWt8NzU4MDQ0Nnw1NTE4MjI0Njk=>.
NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>




--
View this message in context: http://zookeeper-user.578899.n2.nabble.com/Latency-in-asynchronous-mode-tp7580446p7580467.html
Sent from the zookeeper-user mailing list archive at Nabble.com.
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message