hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 张铎 <palomino...@gmail.com>
Subject Re: DISCUSS: Protobufs?
Date Wed, 03 Feb 2016 02:38:38 GMT
Since they already implement an unsafeWrap method which breaks immutability
so add DBB support for CIS does not break anything else. I think it is easy
to make the google guys accept the PR. Let's do it :)

2016-02-03 5:58 GMT+08:00 Stack <stack@duboce.net>:

> Looks like the boys from pb are doing COS only, not CIS, and suggest pull
> request. I'll have a go unless someone else wants to.
> St.Ack
>
> On Tue, Feb 2, 2016 at 11:41 AM, Enis Söztutar <enis.soz@gmail.com> wrote:
>
> > Google guys over at
> > https://github.com/grpc/grpc-java/issues/1054#issuecomment-147295224 are
> > saying that CIS changes may be coming to 2.x from what I understand. If
> so,
> > our life would be easier. Even so, I'm 100% sure we have to do shading
> > since Hadoop will not change it's PB dependency anytime soon.
> >
> > We have to do this before doing shading:
> > https://issues.apache.org/jira/browse/HBASE-15174
> >
> > Enis
> >
> > On Tue, Feb 2, 2016 at 8:15 AM, Stack <stack@duboce.net> wrote:
> >
> > > Thanks Duo. If proto3 had what we wanted, you are suggesting we might
> > move
> > > to proto3 setting it to do proto2 support and shade it so we don't
> clash
> > > with other includes of pb?
> > >
> > > Regards Anoop comment, the note on the end of this issue looks
> promising
> > > but I don't know when it'd see the light of day:
> > > https://github.com/grpc/grpc-java/issues/1054#issuecomment-147295224
> > >
> > > St.Ack
> > >
> > >
> > > On Mon, Feb 1, 2016 at 10:49 PM, Anoop John <anoop.hbase@gmail.com>
> > wrote:
> > >
> > > > UnsafeByteStrings - This may help us to avoid copy even with out our
> > > > HBaseZeroCopyByteString stuff.  But with a DirectByteBuffer, it has
> to
> > > copy
> > > > data to onheap byte[].   We even want a DBB backing !
> > > >
> > > > -Anoop-
> > > >
> > > > On Tue, Feb 2, 2016 at 12:07 PM, 张铎 <palomino219@gmail.com>
wrote:
> > > >
> > > > > https://groups.google.com/forum/#!topic/protobuf/wAqvtPLBsE8
> > > > >
> > > > > PB2 and PB3 are wire compatible, and of course, protobuf-java is
> not
> > > > > compatible so dependency will be a problem... But I think the
> shaded
> > > > client
> > > > > and server can solve the problem?
> > > > >
> > > > > Thanks.
> > > > >
> > > > > 2016-02-02 14:27 GMT+08:00 Stack <stack@duboce.net>:
> > > > >
> > > > > > We are running into a few issues with protobufs.
> > > > > >
> > > > > > + PB always copies all data before making a Message. This
> generates
> > > > > garbage
> > > > > > unnecessarily.
> > > > > > + CodedInputStream does not support ByteBuffers in 2.5. In 2.6
it
> > > does
> > > > > but
> > > > > > again, copies the data out of the BB always; this is especially
> > > painful
> > > > > > when the BB is a DBB with its data offheap and intent is to
keep
> > data
> > > > > > offheap.
> > > > > >
> > > > > > There are other issues. CIS allocates 4k buffers regardless
(See
> > > > > > HBASE-15177).
> > > > > > And then there was the HBaseZeroCopyByteString fun and games
we
> > had a
> > > > > while
> > > > > > back.
> > > > > >
> > > > > > 3.0 PB adds UnsafeByteStrings so can do zero copy. Thats good.
> But
> > > PB3
> > > > is
> > > > > > incompatible with PB2 (or at least, it looks like PB2 clients
> can't
> > > > talk
> > > > > to
> > > > > > PB3 [1]).
> > > > > >
> > > > > > There is javanano protobufs. All is open access, but it too
looks
> > > > > different
> > > > > > to PB2 (i've not tried it).
> > > > > >
> > > > > > Protostuff seems really quiet these times [2].
> > > > > >
> > > > > > Fork (and shade)?
> > > > > >
> > > > > > Thoughts?
> > > > > >
> > > > > > St.Ack
> > > > > >
> > > > > > 1. https://github.com/google/protobuf/releases
> > > > > > 2. https://groups.google.com/forum/#!forum/protostuff
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message