bookkeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "li.penghui"<li.peng...@zhaopin.com.cn>
Subject Re: latency of bookkeeper
Date Mon, 09 Jul 2018 03:35:48 GMT
Thanks for your reply


I try to set different bookkeeper configs


1.default config(i just set zookeeper connection string)


to addEntry sync with 8 threads. i got 5000 ops/s. and can’t improve throughput by incr
threads. 16 threads got 5000 ops/s.


2.journalSyncData = false,journalAdaptiveGroupWrites = false


  i think in this config, i will got lowlatency by one thread. but i test it, i got 400 ops/s
per thread.


so i have two problem with i use bookeeper. i must blocked to wait until bookkeeper tell me
entry id. if use aync apis, i can’t return add result to users, because rpc framework in
my company can’t support async calls.


1.how can i get lowlatency in one thread.
2.how can i get highthroughput in one ledger with muti threads and sync apis.




原始邮件
发件人:li.penghuili.penghui@zhaopin.com.cn
收件人:useruser@bookkeeper.apache.org
发送时间:2018年7月6日(周五) 17:56
主题:Re: latency of bookkeeper


Can I close the WAL if i can tolerate any enries lost. if WAL can be closed, i can use bookkeeper
in different scenes.


原始邮件
发件人:Sijie Guoguosijie@gmail.com
收件人:useruser@bookkeeper.apache.org
发送时间:2018年7月6日(周五) 16:11
主题:Re: latency of bookkeeper


I think your question is a bit not clear, latency and throughput are two kind of different
metrics. Your question seems to be asking for high throughput.


Anyway, I will try to explain the performance tradeoff between latency and throughput and
hope that helps.


Bookkeeper by default fsync the data to disks. It does 1ms group commit by default to keep
a good tradeoff between throughput and latency.


1) if you are using synchronous adds, since you are blocking on waiting write response, your
single thread throughput will be limited by the group commit interval.
You can use multiple threads to improve throughput, since multiple writes will be grouped
together writing to disks, you will get as lower latency as your group commit interval for
your writes.


2) if your application can leverage asynchronous adds, you should consider using asynchronous
apis. It will offer you the best latency while be able to achieve high throughput.


3) the latency will eventually be dominated by your disk fsync latency. SSD or HDD with battery
will have good fsync latency (about half millisecond). However if your disk is not as good
to
provide such lower latency, you can consider disable fsync and rely on replication to achieve
durability.https://github.com/apache/bookkeeper/blob/master/conf/bk_server.conf#L309


Hope this helps.


On Thu, Jul 5, 2018 at 11:46 PM li.penghui@zhaopin.com.cn li.penghui@zhaopin.com.cn wrote:



  Hi


    I try to use bookkeeper. i care latency of write. so start a test in single thread. get
400 ops/s in double SSD.
  
    how to improve performance to get the low-latency.


  Thanks.


li.penghui@zhaopin.com.cn
Can I close the WAL if i can tolerate any enries lost. if WAL can be closed, i can use bookkeeper
in different scenes.


原始邮件
发件人:Sijie Guoguosijie@gmail.com
收件人:useruser@bookkeeper.apache.org
发送时间:2018年7月6日(周五) 16:11
主题:Re: latency of bookkeeper


I think your question is a bit not clear, latency and throughput are two kind of different
metrics. Your question seems to be asking for high throughput.


Anyway, I will try to explain the performance tradeoff between latency and throughput and
hope that helps.


Bookkeeper by default fsync the data to disks. It does 1ms group commit by default to keep
a good tradeoff between throughput and latency.


1) if you are using synchronous adds, since you are blocking on waiting write response, your
single thread throughput will be limited by the group commit interval.
You can use multiple threads to improve throughput, since multiple writes will be grouped
together writing to disks, you will get as lower latency as your group commit interval for
your writes.


2) if your application can leverage asynchronous adds, you should consider using asynchronous
apis. It will offer you the best latency while be able to achieve high throughput.


3) the latency will eventually be dominated by your disk fsync latency. SSD or HDD with battery
will have good fsync latency (about half millisecond). However if your disk is not as good
to
provide such lower latency, you can consider disable fsync and rely on replication to achieve
durability.https://github.com/apache/bookkeeper/blob/master/conf/bk_server.conf#L309


Hope this helps.


On Thu, Jul 5, 2018 at 11:46 PM li.penghui@zhaopin.com.cn li.penghui@zhaopin.com.cn wrote:



  Hi


    I try to use bookkeeper. i care latency of write. so start a test in single thread. get
400 ops/s in double SSD.
  
    how to improve performance to get the low-latency.


  Thanks.


li.penghui@zhaopin.com.cn
Mime
View raw message