bookkeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Flavio Junqueira <...@apache.org>
Subject Re: Fastest way to write to bookkeeper
Date Mon, 14 Sep 2015 17:24:10 GMT
Ok, so that's 50k per write and you seem to be getting 250 kbytes per second. That's low, you
should be able to get higher throughput. We used to get over 20k adds/s of 1kbytes each, which
is more like 20Mbytes/s.

-Flavio
  
> On 14 Sep 2015, at 06:49, Enrico Olivelli - Diennea <enrico.olivelli@diennea.com>
wrote:
> 
> I did some benchmark and actually writing a batch of 100 (little, 500 bytes) entries
using asyncAddEntry is as fast as writing only one entry, that is 200 ms.
>  
> I will resolve my problems trying to avoid to single entry writes
>  
> Thanks
>  
> Enrico Olivelli
>  
> Da: Ivan Kelly [mailto:ivank@apache.org] 
> Inviato: lunedì 14 settembre 2015 15:46
> A: user@bookkeeper.apache.org
> Oggetto: Re: Fastest way to write to bookkeeper
>  
> May I suggest, before making any code changes, you measure the difference in MB/s between
large writes and small writes? I do recall there was some advantage to using large entries
in the past, more than 1k, but I don't remember what it was, and it may not still be true.
A lot of code has changed since then.
> 
> In theory, anything greater than the MTU shouldn't give too much of a boost.
> 
> -Ivan
>  
> On Mon, Sep 14, 2015 at 3:35 PM Flavio Junqueira <fpj@apache.org <mailto:fpj@apache.org>>
wrote:
> Hi Enrico,
>  
> What's the size of each entry? If they are small say just a few bytes, then you're indeed
better off grouping them. If they are 1k or more, then the benefit of grouping shouldn't be
much.
>  
> About extending the API, the only disadvantage I can see of grouping writes into an entry
rather than writing a batch of entries is that a read request will have to read them all.
I personally don't like so much the idea of a batch call because it makes the code a bit messier.
You need to start a batch, add a bunch of stuff, flush the batch, start a new batch, add a
bunch of stuff, and so on. With addEntry, you just invoke it every time you have a new message.
>  
> -Flavio
>  
> On 14 Sep 2015, at 05:02, Enrico Olivelli - Diennea <enrico.olivelli@diennea.com <mailto:enrico.olivelli@diennea.com>>
wrote:
>  
> Hi,
> What is the fastest way to write to BookKeeper a batch of entries ?
>  
> I’m using a sequence of asyncAddEntry, some thing like the code below:
>  
> List<Long> res= new ArrayList<>(); // holds entry sequence numbers
> CountDownLatch latch = new CountDownLatch(size);
> for (int i = 0; i < size; i++) {
> …..
>    this.out.asyncAddEntry(entry, new AsyncCallback.AddCallback() {
>  
>     public void addComplete(int rc, LedgerHandle lh, long entryId, Object i) {
>                             int index = (Integer) i;
>                             if (rc != BKException.Code.OK) {
>                                 BKException error = BKException.create(rc);
>                                 exception.value = error;
>                                 res.set(index, null);
>                                 for (int j = 0; j < size; j++) {
>                                     // early exit
>                                     latch.countDown();
>                                 }
>                             } else {
>                                 res.set(index, entryId);
>                                 latch.countDown();
>                             }
>                         }
>                     }, i);
>     }
> latch.await();
>  
> Would it be faster to group all the entries in one “large” entry ? This may alter
application semantics but if it would be faster I will do the refactor
>  
> Can I file an issue in order to implement a “batchAddEntries” which implements the
write of a batch of entries within the native Bookkeeper client ?
>  
>  
>  
>  
>  
> Enrico Olivelli 
> Software Development Manager @Diennea 
> Tel.: (+39) 0546 066100 - Int. 925 
> Viale G.Marconi 30/14 - 48018 Faenza (RA) 
> 
> MagNews - E-mail Marketing Solutions
> http://www.magnews.it <http://www.magnews.it/>
> Diennea - Digital Marketing Solutions
> http://www.diennea.com <http://www.diennea.com/>
>  
>  
> Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed email marketing!
http://www.magnews.it/newsletter/ <http://www.magnews.it/newsletter/>
>  
> 
> Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed email marketing!
http://www.magnews.it/newsletter/ <http://www.magnews.it/newsletter/>

Mime
View raw message