bookkeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sijie Guo <guosi...@gmail.com>
Subject Re: Re: Re: hi, why write speed of bookkeeper-4.3.0 is very slow?
Date Sun, 19 Oct 2014 04:52:03 GMT
I really doubt the number 2941 entries/s here. since your test application
is writing entries sequentially, so it means each write request need to be
completed in 0.3ms, which doesn't seem to be reasonable per my knowledge.
Could you point what is your disk characteristics?

Besides that, I think your test is really biased.

- 10/100/1000 entries is too small, which most of all entries will be just
in memory for ledger storage. you will never know how's the impact on
latency if ledger storage is doing disk I/O.
- since bookkeeper is targeting a durable journal, all requests are needed
to be synced to disk. the latency of adding entry is around millisecond
level. the way you are testing bookie isn't correct. since the latency of
an entry include network RTTs and disk fsync time, if there is an addEntry
operation encountered network/disk hiccups, you will definitely see lower
entries be sent in one second.
- we encourage people to use asynchronous API to addEntry in a real system
for latency and throughput.

I would suggest:

* try using bookkeeper-benchmark/bin/benchmark to benchmark your bookie
cluster using different bookkeeper versions, to see what is the difference.
* if you are going to using you test benchmark, try running your benchmark
using a larger entries, to get your benchmark run enough time.
* in 4.3.0 bookkeeper, try choose one stats provider and set
statsProviderClass in your bookie configuration. so you could get metrics
about your journal operations during benchmarking, so we will know what is
slow in your system.


Last one, there are several journal settings introduced in bookkeeper
4.3.0. You could use different combination to support different
characterized workloads.
http://zookeeper.apache.org/bookkeeper/docs/r4.3.0/bookieConfigParams.html

In our systems, we usually set journalAdaptiveGroupWrites =
true, journalFlushWhenQueueEmpty = true and journalMaxGroupWaitMSec to a
value that meet your latency SLA. Setting journalFlushWhenQueueEmpty to
true, it would make the number of flushes more sustainable when bursty
traffic happened; setting journalAdaptiveGroupWrites to true would try
grouping flushes together into one fsync, which reduce number of fsyncs to
your disk. Choosing a group commit time is a tradeoff between latency and
throughput.

- Sijie







On Fri, Oct 17, 2014 at 8:28 AM, sky_cord <sky_cord@163.com> wrote:

> Thanks for your help.
> After adding the two config items into bk_server.conf :
>
> journalFlushWhenQueueEmpty=true
> journalMaxGroupWaitMSec=0
>
> The write speed rise up to 800-900 entries per second in one ledger, length of each entry
is 50 bytes.
>
> But it's still lower than I expected(The sync write  speed is 2941 entries/s based
onbookkeepr-4.2.2).
>
> Could you describe your test result?   And Is there any way to improve write performance
?
>
> Thank you.
>
>
>
>
> At 2014-10-17 20:13:07, "Ivan Kelly" <ivank@apache.org> wrote:
> >Ah, I know what this is. Twitter added some changes that makes write
> >throughput much higher when there's a high write rate. In this case
> >the latency for a single writes goes go up though (as it's doing more
> >buffering on the journal). Try setting:
> >
> >journalFlushWhenQueueEmpty=true
> >journalMaxGroupWaitMSec=0
> >
> >in bk_server.conf
> >
> >
> >Also, the journal and ledger directories are on the same disk. This
> >will have a detrimental affect once the write buffers start flushing.
> >We always recommend that you put the journal on its own independent
> >disk.
> >
> >The reads are fast because it doesn't need to hit disk, as there's not
> >much data that has been written.
> >
> >-Ivan
> >
> >On 17 October 2014 13:32, sky_cord@163.com <sky_cord@163.com> wrote:
> >> The write speed of bookkeper that version less than 4.30 is bigger than 1200
> >> entries per second when i use the same test code
> >>
> >> ________________________________
> >> sky_cord@163.com
> >>
> >>
> >> From: Jiannan Wang
> >> Date: 2014-10-17 18:31
> >> To: sky_cord@163.com
> >> CC: bookkeeper-user@zookeeper.apache.org
> >> Subject: Re: hi, why write speed of bookkeeper-4.3.0 is very slow?
> >> Could you share more information:
> >>    - What's the number of write_num?
> >>    - What's the read throughput under your test?
> >>    - Does the 5 bookie servers run in same machine?
> >>    - Could you check whether StringUtil.div works well?
> >>    - Could you also try async write API and tell us the output?
> >>
> >> Regards,
> >> Jiannan
> >>
> >> From: "sky_cord@163.com" <sky_cord@163.com>
> >> Reply-To: "bookkeeper-user@zookeeper.apache.org"
> >> <bookkeeper-user@zookeeper.apache.org>
> >> Date: Friday, October 17, 2014 at 3:38 PM
> >> To: bookkeeper-user <bookkeeper-user@zookeeper.apache.org>
> >> Subject: hi, why write speed of bookkeeper-4.3.0 is very slow?
> >>
> >> hi,
> >>      I have tried bookkeeper-4.3.0 today. But its write speed is 4 entries
> >> per sec in my environment.
> >>      There are 5 bookkeeper-server-4.3.0 in my env.
> >>      Is there anything wrong? Test codes as follows:
> >> public void syncWriteLedger(long write_num) throws InterruptedException,
> >> BKException {
> >> byte[] pwd = "foobar".getBytes();
> >> LedgerHandle lh = client.createLedger(3, 2,
> >> DigestType.MAC, pwd);
> >>
> >> long ledgerId = lh.getId();
> >> System.out.println("ledgerId=" + ledgerId);
> >> System.out
> >> .println("data_len="
> >> + ("Hello World!afdafdafdfxtyb457u8n9,7.,pbesgcerfqx" + write_num)
> >> .getBytes().length);
> >> byte[] data = ("Hello World!afdafdafdfxtyb457u8n9,7.,pbesgcerfqx" +
> >> write_num).getBytes();
> >> long st = System.currentTimeMillis();
> >> for (int i = 0; i < write_num; i++) {
> >> lh.addEntry(data);
> >> }
> >> long ut = System.currentTimeMillis() - st;
> >> lh.close();
> >> double uts = StringUtil.div(ut, 1000, 3);
> >> double wspeed = StringUtil.div(write_num, uts, 2);
> >> System.out.println("ut=" + ut + " ms, write_num=" + write_num
> >> + ", write speed=" + wspeed + "/s");
> >>
> >>
> >> LedgerHandle lh2 = client.openLedger(ledgerId, BookKeeper.DigestType.MAC,
> >> pwd);
> >> long lastEntry = lh2.getLastAddConfirmed();
> >> st = System.currentTimeMillis();
> >> Enumeration<LedgerEntry> entries = lh2.readEntries(0, lastEntry);
> >> while (entries.hasMoreElements()) {
> >> byte[] bytes = entries.nextElement().getEntry();
> >> System.out.println(new String(bytes));
> >> }
> >> ut = System.currentTimeMillis() - st;
> >> lh2.close();
> >> uts = StringUtil.div(ut, 1000, 3);
> >> double rspeed = StringUtil.div(write_num, uts, 2);
> >> System.out.println("ut=" + ut + " ms, read_num=" + write_num
> >> + ", read speed=" + rspeed + "/s");
> >>
> >> }
> >> ________________________________
> >> sky_cord@163.com
>
>
>
>

Mime
View raw message