hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anoop John <anoop.hb...@gmail.com>
Subject Re: DISCUSS: Protobufs?
Date Tue, 02 Feb 2016 06:49:34 GMT
UnsafeByteStrings - This may help us to avoid copy even with out our
HBaseZeroCopyByteString stuff.  But with a DirectByteBuffer, it has to copy
data to onheap byte[].   We even want a DBB backing !

-Anoop-

On Tue, Feb 2, 2016 at 12:07 PM, 张铎 <palomino219@gmail.com> wrote:

> https://groups.google.com/forum/#!topic/protobuf/wAqvtPLBsE8
>
> PB2 and PB3 are wire compatible, and of course, protobuf-java is not
> compatible so dependency will be a problem... But I think the shaded client
> and server can solve the problem?
>
> Thanks.
>
> 2016-02-02 14:27 GMT+08:00 Stack <stack@duboce.net>:
>
> > We are running into a few issues with protobufs.
> >
> > + PB always copies all data before making a Message. This generates
> garbage
> > unnecessarily.
> > + CodedInputStream does not support ByteBuffers in 2.5. In 2.6 it does
> but
> > again, copies the data out of the BB always; this is especially painful
> > when the BB is a DBB with its data offheap and intent is to keep data
> > offheap.
> >
> > There are other issues. CIS allocates 4k buffers regardless (See
> > HBASE-15177).
> > And then there was the HBaseZeroCopyByteString fun and games we had a
> while
> > back.
> >
> > 3.0 PB adds UnsafeByteStrings so can do zero copy. Thats good. But PB3 is
> > incompatible with PB2 (or at least, it looks like PB2 clients can't talk
> to
> > PB3 [1]).
> >
> > There is javanano protobufs. All is open access, but it too looks
> different
> > to PB2 (i've not tried it).
> >
> > Protostuff seems really quiet these times [2].
> >
> > Fork (and shade)?
> >
> > Thoughts?
> >
> > St.Ack
> >
> > 1. https://github.com/google/protobuf/releases
> > 2. https://groups.google.com/forum/#!forum/protostuff
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message