hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: DISCUSS: Protobufs?
Date Tue, 02 Feb 2016 16:15:46 GMT
Thanks Duo. If proto3 had what we wanted, you are suggesting we might move
to proto3 setting it to do proto2 support and shade it so we don't clash
with other includes of pb?

Regards Anoop comment, the note on the end of this issue looks promising
but I don't know when it'd see the light of day:
https://github.com/grpc/grpc-java/issues/1054#issuecomment-147295224

St.Ack


On Mon, Feb 1, 2016 at 10:49 PM, Anoop John <anoop.hbase@gmail.com> wrote:

> UnsafeByteStrings - This may help us to avoid copy even with out our
> HBaseZeroCopyByteString stuff.  But with a DirectByteBuffer, it has to copy
> data to onheap byte[].   We even want a DBB backing !
>
> -Anoop-
>
> On Tue, Feb 2, 2016 at 12:07 PM, 张铎 <palomino219@gmail.com> wrote:
>
> > https://groups.google.com/forum/#!topic/protobuf/wAqvtPLBsE8
> >
> > PB2 and PB3 are wire compatible, and of course, protobuf-java is not
> > compatible so dependency will be a problem... But I think the shaded
> client
> > and server can solve the problem?
> >
> > Thanks.
> >
> > 2016-02-02 14:27 GMT+08:00 Stack <stack@duboce.net>:
> >
> > > We are running into a few issues with protobufs.
> > >
> > > + PB always copies all data before making a Message. This generates
> > garbage
> > > unnecessarily.
> > > + CodedInputStream does not support ByteBuffers in 2.5. In 2.6 it does
> > but
> > > again, copies the data out of the BB always; this is especially painful
> > > when the BB is a DBB with its data offheap and intent is to keep data
> > > offheap.
> > >
> > > There are other issues. CIS allocates 4k buffers regardless (See
> > > HBASE-15177).
> > > And then there was the HBaseZeroCopyByteString fun and games we had a
> > while
> > > back.
> > >
> > > 3.0 PB adds UnsafeByteStrings so can do zero copy. Thats good. But PB3
> is
> > > incompatible with PB2 (or at least, it looks like PB2 clients can't
> talk
> > to
> > > PB3 [1]).
> > >
> > > There is javanano protobufs. All is open access, but it too looks
> > different
> > > to PB2 (i've not tried it).
> > >
> > > Protostuff seems really quiet these times [2].
> > >
> > > Fork (and shade)?
> > >
> > > Thoughts?
> > >
> > > St.Ack
> > >
> > > 1. https://github.com/google/protobuf/releases
> > > 2. https://groups.google.com/forum/#!forum/protostuff
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message