bookkeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ivan Kelly <iv...@apache.org>
Subject Re: Re: hi, why write speed of bookkeeper-4.3.0 is very slow?
Date Fri, 17 Oct 2014 12:13:07 GMT
Ah, I know what this is. Twitter added some changes that makes write
throughput much higher when there's a high write rate. In this case
the latency for a single writes goes go up though (as it's doing more
buffering on the journal). Try setting:

journalFlushWhenQueueEmpty=true
journalMaxGroupWaitMSec=0

in bk_server.conf


Also, the journal and ledger directories are on the same disk. This
will have a detrimental affect once the write buffers start flushing.
We always recommend that you put the journal on its own independent
disk.

The reads are fast because it doesn't need to hit disk, as there's not
much data that has been written.

-Ivan

On 17 October 2014 13:32, sky_cord@163.com <sky_cord@163.com> wrote:
> The write speed of bookkeper that version less than 4.30 is bigger than 1200
> entries per second when i use the same test code
>
> ________________________________
> sky_cord@163.com
>
>
> From: Jiannan Wang
> Date: 2014-10-17 18:31
> To: sky_cord@163.com
> CC: bookkeeper-user@zookeeper.apache.org
> Subject: Re: hi, why write speed of bookkeeper-4.3.0 is very slow?
> Could you share more information:
>    - What's the number of write_num?
>    - What's the read throughput under your test?
>    - Does the 5 bookie servers run in same machine?
>    - Could you check whether StringUtil.div works well?
>    - Could you also try async write API and tell us the output?
>
> Regards,
> Jiannan
>
> From: "sky_cord@163.com" <sky_cord@163.com>
> Reply-To: "bookkeeper-user@zookeeper.apache.org"
> <bookkeeper-user@zookeeper.apache.org>
> Date: Friday, October 17, 2014 at 3:38 PM
> To: bookkeeper-user <bookkeeper-user@zookeeper.apache.org>
> Subject: hi, why write speed of bookkeeper-4.3.0 is very slow?
>
> hi,
>      I have tried bookkeeper-4.3.0 today. But its write speed is 4 entries
> per sec in my environment.
>      There are 5 bookkeeper-server-4.3.0 in my env.
>      Is there anything wrong? Test codes as follows:
> public void syncWriteLedger(long write_num) throws InterruptedException,
> BKException {
> byte[] pwd = "foobar".getBytes();
> LedgerHandle lh = client.createLedger(3, 2,
> DigestType.MAC, pwd);
>
> long ledgerId = lh.getId();
> System.out.println("ledgerId=" + ledgerId);
> System.out
> .println("data_len="
> + ("Hello World!afdafdafdfxtyb457u8n9,7.,pbesgcerfqx" + write_num)
> .getBytes().length);
> byte[] data = ("Hello World!afdafdafdfxtyb457u8n9,7.,pbesgcerfqx" +
> write_num).getBytes();
> long st = System.currentTimeMillis();
> for (int i = 0; i < write_num; i++) {
> lh.addEntry(data);
> }
> long ut = System.currentTimeMillis() - st;
> lh.close();
> double uts = StringUtil.div(ut, 1000, 3);
> double wspeed = StringUtil.div(write_num, uts, 2);
> System.out.println("ut=" + ut + " ms, write_num=" + write_num
> + ", write speed=" + wspeed + "/s");
>
>
> LedgerHandle lh2 = client.openLedger(ledgerId, BookKeeper.DigestType.MAC,
> pwd);
> long lastEntry = lh2.getLastAddConfirmed();
> st = System.currentTimeMillis();
> Enumeration<LedgerEntry> entries = lh2.readEntries(0, lastEntry);
> while (entries.hasMoreElements()) {
> byte[] bytes = entries.nextElement().getEntry();
> System.out.println(new String(bytes));
> }
> ut = System.currentTimeMillis() - st;
> lh2.close();
> uts = StringUtil.div(ut, 1000, 3);
> double rspeed = StringUtil.div(write_num, uts, 2);
> System.out.println("ut=" + ut + " ms, read_num=" + write_num
> + ", read speed=" + rspeed + "/s");
>
> }
> ________________________________
> sky_cord@163.com

Mime
View raw message