bookkeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ivan Kelly <iv...@apache.org>
Subject Re: Fastest way to write to bookkeeper
Date Mon, 14 Sep 2015 13:46:01 GMT
May I suggest, before making any code changes, you measure the difference
in MB/s between large writes and small writes? I do recall there was some
advantage to using large entries in the past, more than 1k, but I don't
remember what it was, and it may not still be true. A lot of code has
changed since then.

In theory, anything greater than the MTU shouldn't give too much of a boost.

-Ivan

On Mon, Sep 14, 2015 at 3:35 PM Flavio Junqueira <fpj@apache.org> wrote:

> Hi Enrico,
>
> What's the size of each entry? If they are small say just a few bytes,
> then you're indeed better off grouping them. If they are 1k or more, then
> the benefit of grouping shouldn't be much.
>
> About extending the API, the only disadvantage I can see of grouping
> writes into an entry rather than writing a batch of entries is that a read
> request will have to read them all. I personally don't like so much the
> idea of a batch call because it makes the code a bit messier. You need to
> start a batch, add a bunch of stuff, flush the batch, start a new batch,
> add a bunch of stuff, and so on. With addEntry, you just invoke it every
> time you have a new message.
>
> -Flavio
>
> On 14 Sep 2015, at 05:02, Enrico Olivelli - Diennea <
> enrico.olivelli@diennea.com> wrote:
>
> Hi,
> What is the fastest way to write to BookKeeper a batch of entries ?
>
> I’m using a sequence of asyncAddEntry, some thing like the code below:
>
> List<Long> res= new ArrayList<>(); // holds entry sequence numbers
> CountDownLatch latch = new CountDownLatch(size);
> for (int i = 0; i < size; i++) {
> …..
>    this.out.asyncAddEntry(entry, new AsyncCallback.AddCallback() {
>
>     public void addComplete(int rc, LedgerHandle lh, long entryId, Object
> i) {
>                             int index = (Integer) i;
>                             if (rc != BKException.Code.OK) {
>                                 BKException error = BKException.create(rc);
>                                 exception.value = error;
>                                 res.set(index, null);
>                                 for (int j = 0; j < size; j++) {
>                                     // early exit
>                                     latch.countDown();
>                                 }
>                             } else {
>                                 res.set(index, entryId);
>                                 latch.countDown();
>                             }
>                         }
>                     }, i);
>     }
> latch.await();
>
> Would it be faster to group all the entries in one “large” entry ? This
> may alter application semantics but if it would be faster I will do the
> refactor
>
> Can I file an issue in order to implement a “batchAddEntries” which
> implements the write of a batch of entries within the native Bookkeeper
> client ?
>
>
>
>
>
> *Enrico Olivelli*
> Software Development Manager @Diennea
> Tel.: (+39) 0546 066100 - Int. 925
> Viale G.Marconi 30/14 - 48018 Faenza (RA)
>
> MagNews - E-mail Marketing Solutions
> http://www.magnews.it
> Diennea - Digital Marketing Solutions
> http://www.diennea.com
>
>
> ------------------------------
> Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed
> email marketing! http://www.magnews.it/newsletter/
>
>
>

Mime
View raw message