Return-Path: X-Original-To: apmail-bookkeeper-user-archive@www.apache.org Delivered-To: apmail-bookkeeper-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5E16017915 for ; Wed, 10 Jun 2015 14:45:39 +0000 (UTC) Received: (qmail 14824 invoked by uid 500); 10 Jun 2015 14:45:39 -0000 Delivered-To: apmail-bookkeeper-user-archive@bookkeeper.apache.org Received: (qmail 14778 invoked by uid 500); 10 Jun 2015 14:45:39 -0000 Mailing-List: contact user-help@bookkeeper.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@bookkeeper.apache.org Delivered-To: mailing list user@bookkeeper.apache.org Received: (qmail 14768 invoked by uid 99); 10 Jun 2015 14:45:38 -0000 Received: from Unknown (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 10 Jun 2015 14:45:38 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 8326E1827EA for ; Wed, 10 Jun 2015 14:45:38 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.879 X-Spam-Level: ** X-Spam-Status: No, score=2.879 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id j8SVSJ4rR3Mk for ; Wed, 10 Jun 2015 14:45:37 +0000 (UTC) Received: from mail-wi0-f182.google.com (mail-wi0-f182.google.com [209.85.212.182]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id 6D17820F24 for ; Wed, 10 Jun 2015 14:45:36 +0000 (UTC) Received: by wiga1 with SMTP id a1so50992215wig.0 for ; Wed, 10 Jun 2015 07:45:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=0DV2bACTxPDvJ0lruiq6Efta9Ku7kAXr+uTesb9duZI=; b=er03q8/nmPZIE537uy95QSLDWt5I4OB9Y/NqilIeBe4mGggOlZ42blXFGT8KOB6LZj 0m8HAZi7Ok/0x76zW45XRzWVVFk6tfBDMs6EOIdZ+zhGZQ3UMPf9ouf7hU96+9VPAqK6 /iZWk0Tii2vwPJUupDGqM0uA/SVPvxv3uIeq5ZesIKmWSng5W0iAAb0Pj++7I5nYxW63 OfuSQv0PreqluvM+iazMsnA0gA5HlzeAO5rwBTpL/9bhkOtSzy0Oigw6hLYfLIaMb1D5 LCh2dAJ/mrae4eiNT1cNvybeq8bwzp/wCGKSaLr32lXubAwhUUL2gGsCMVDFB3wI0GX7 Es1Q== X-Received: by 10.180.91.137 with SMTP id ce9mr9017316wib.76.1433947530254; Wed, 10 Jun 2015 07:45:30 -0700 (PDT) MIME-Version: 1.0 Received: by 10.180.7.232 with HTTP; Wed, 10 Jun 2015 07:45:09 -0700 (PDT) In-Reply-To: References: <220517564.205637.1433939265291.JavaMail.yahoo@mail.yahoo.com> From: Aniruddha Laud Date: Wed, 10 Jun 2015 07:45:09 -0700 Message-ID: Subject: Re: Low write bandwidth To: user@bookkeeper.apache.org Cc: Flavio Junqueira Content-Type: multipart/alternative; boundary=f46d043749c52ae0d805182aedb4 --f46d043749c52ae0d805182aedb4 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On Wed, Jun 10, 2015 at 7:00 AM, Maciej Smole=C5=84ski = wrote: > Thank You for Your comment. > > Unfortunately, these option will not help in my case. > In my case BookKeeper client will receive next request when previous > request is confirmed. > It is expected also that there will be only single stream of such request= s. > > I would like to understand how to achieve performance equal to the networ= k > bandwidth. > to saturate bandwidth, you will have to have more than one outstanding request. 250 requests/second gives you 4ms per request. With each entry 100K in size, that's not unreasonable. My suggestion would be to monitor the write latency from the client to the server. ping -s 65000 should give you a baseline for what to expect with latencies. With 100K packets, you are going to see fragmentation at both the IP and the Ethernet layer. That wasn't the case with 1K payload. How many hops does one need to go from one machine to another? - higher the hops, higher the latency > > > On Wed, Jun 10, 2015 at 2:27 PM, Flavio Junqueira > wrote: > >> BK currently isn't wired to stream bytes to a ledger, so writing >> synchronously large entries as you're doing is likely not to get the bes= t >> its performance. A couple of things you could try to get higher performa= nce >> are to write asynchronously and to have multiple clients writing. >> >> -Flavio >> >> >> >> >> On Wednesday, June 10, 2015 12:08 PM, Maciej Smole=C5=84ski < >> jezdnia@gmail.com> wrote: >> >> >> >> Hello, >> >> I'm testing BK performance when appending 100K entries synchronously fro= m >> 1 thread (using one ledger). >> The performance I get is 250 entries/s. >> >> What performance should I expect ? >> >> My setup: >> >> Ledger: >> Ensemble size: 3 >> Quorum size: 2 >> >> 1 client machine and 3 server machines. >> >> Network: >> Each machine with bonding: 4 x 1000Mbps on each machine >> manually tested between client and server: 400MB/s >> >> Disk: >> I tested two configurations: >> dedicated disks with ext3 (different for zookeeper, journal, data, index= , >> log) >> dedicated ramfs partitions (different for zookeeper, journal, data, >> index, log) >> >> In both configurations the performance is the same: 250 entries / s (25M= B >> / s). >> I confirmed this with measured network bandwidth: >> - on client 50 MB/s >> - on server 17 MB/s >> >> I run java with profiler enabled on BK client and BK server but didn't >> find anything unexpected (but I don't know bookkeeper internals). >> >> I tested it with two BookKeeper versions: >> - 4.3.0 >> - 4.2.2 >> The result were the same with both BookKeeper versions. >> >> What should be changed/checked to get better performance ? >> >> Kind regards, >> Maciej >> >> >> >> >> >> >> >> >> >> >> > --f46d043749c52ae0d805182aedb4 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable


On Wed, Jun 10, 2015 at 7:00 AM, Maciej Smole=C5=84ski <jezdnia@gmail.= com> wrote:
Thank You for Your comment.

Unfortunately, these option= will not help in my case.
In my case BookKeeper= client will receive next request when previous request is confirmed.
It is expected also that there will be only single st= ream of such requests.

I would like to under= stand how to achieve performance equal to the network bandwidth.
=
=C2=A0
to saturate bandwidth, you w= ill have to have more than one outstanding request. 250 requests/second giv= es you 4ms per request. With each entry 100K in size, that's not unreas= onable. My suggestion would be to monitor the write latency from the client= to the server.=C2=A0

ping -s 65000 should give yo= u a baseline for what to expect with latencies.=C2=A0

<= div>With 100K packets, you are going to see fragmentation at both the IP an= d the Ethernet layer. That wasn't the case with 1K payload.=C2=A0
=

How many hops does one need to go from one machine to a= nother? - higher the hops, higher the latency


=C2=A0

On Wed= , Jun 10, 2015 at 2:27 PM, Flavio Junqueira <fpjunqueira@yahoo.com> wrote:
BK currently isn't wired to stream bytes to a ledger, so writing syn= chronously large entries as you're doing is likely not to get the best = its performance. A couple of things you could try to get higher performance= are to write asynchronously and to have multiple clients writing.=C2=A0

<= /div>
-Flavio




On Wednesday, June = 10, 2015 12:08 PM, Maciej Smole=C5=84ski <jezdnia@gmail.com> wrote:


Hello,

I'm testing BK performance when appending 10= 0K entries synchronously from 1 thread (using one ledger).
Th= e performance I get is 250 entries/s.

What per= formance should I expect ?

My setup:
=

Ledger:
Ensemble size: 3
Quorum size: 2

1 client machine and 3 server machines.

Network:
Each machine with bonding: 4 x 1000Mbps on each machine
<= /div>
manually tested between client and server: 400MB/s

<= div>Disk:
I tested two configurations:
dedicate= d disks with ext3 (different for zookeeper, journal, data, index, log)
<= /div>
dedicated ramfs partitions (different for zookeeper, journal, dat= a, index, log)

In both configurations the performance is = the same: 250 entries / s (25MB / s).
I confirmed this with m= easured network bandwidth:
- on client 50 MB/s
= - on server 17 MB/s

I run java with profiler enabled on B= K client and BK server but didn't find anything unexpected (but I don&#= 39;t know bookkeeper internals).

I tested it with two Boo= kKeeper versions:
- 4.3.0
- 4.2.2
The result were the s= ame with both BookKeeper versions.

What should be changed= /checked to get better performance ?

Kind regards,
Maciej





=C2= =A0




<= /div>


--f46d043749c52ae0d805182aedb4--