Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAMUu0w_Q0-aGuMQPN1qDwUZ1fJkkrFpwp+L51+rO7q5u=Gh1og@mail.gmail.com>
References: 
 <CADcMMgGQRDQ7WBSSQTEwPHp3bn1H+aHVBTBBqdKjyj3PBHwYkg@mail.gmail.com>
 <CAAAYAnNQWDFmC5yQyMmX6Z2ZjCsf3x9ONfqcqOqfzWQ2B48hmw@mail.gmail.com>
 <CAOtJ30oo9aeeShiVckuaVP+Hbn0pUWChuHtQrRWyMqAe0N2joQ@mail.gmail.com>
 <CADcMMgFe+h9bnPvJHfvH=YrT+g=hS4LVXHgsvWDdgWWvaXF4xw@mail.gmail.com>
 <CAMUu0w_Q0-aGuMQPN1qDwUZ1fJkkrFpwp+L51+rO7q5u=Gh1og@mail.gmail.com>
From: =?UTF-8?Q?Enis_S=C3=B6ztutar?= <enis.soz@gmail.com>
Date: Tue, 2 Feb 2016 11:47:24 -0800
Message-ID: 
 <CAMUu0w_Kqvv_KSUjFNnYpaetPx9h3viOGn_BUsOvYyU3_GFN6A@mail.gmail.com>
Subject: Re: DISCUSS: Protobufs?
To: "dev@hbase.apache.org" <dev@hbase.apache.org>
Content-Type: multipart/alternative; boundary=047d7b6725dc6447c3052acec64a

--047d7b6725dc6447c3052acec64a
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

BTW, we should also be looking at https://google.github.io/flatbuffers/ or
https://capnproto.org/ for serialization as an option. The idea is to not
allocate objects and prevent allocations altogether.

We are allocating PB objects for every Get / Put, then we allocate our Get
/ Put objects. At least we can save on 1.

Although, just switching our serialization format again will be a huge
undertaking with obvious wire-incompatibility issues. If PB3 or 2.x gives
us what we want in terms of preventing big byte[] allocations, we would
gain regardless.

Enis

On Tue, Feb 2, 2016 at 11:41 AM, Enis S=C3=B6ztutar <enis.soz@gmail.com> wr=
ote:

> Google guys over at
> https://github.com/grpc/grpc-java/issues/1054#issuecomment-147295224 are
> saying that CIS changes may be coming to 2.x from what I understand. If s=
o,
> our life would be easier. Even so, I'm 100% sure we have to do shading
> since Hadoop will not change it's PB dependency anytime soon.
>
> We have to do this before doing shading:
> https://issues.apache.org/jira/browse/HBASE-15174
>
> Enis
>
> On Tue, Feb 2, 2016 at 8:15 AM, Stack <stack@duboce.net> wrote:
>
>> Thanks Duo. If proto3 had what we wanted, you are suggesting we might mo=
ve
>> to proto3 setting it to do proto2 support and shade it so we don't clash
>> with other includes of pb?
>>
>> Regards Anoop comment, the note on the end of this issue looks promising
>> but I don't know when it'd see the light of day:
>> https://github.com/grpc/grpc-java/issues/1054#issuecomment-147295224
>>
>> St.Ack
>>
>>
>> On Mon, Feb 1, 2016 at 10:49 PM, Anoop John <anoop.hbase@gmail.com>
>> wrote:
>>
>> > UnsafeByteStrings - This may help us to avoid copy even with out our
>> > HBaseZeroCopyByteString stuff.  But with a DirectByteBuffer, it has to
>> copy
>> > data to onheap byte[].   We even want a DBB backing !
>> >
>> > -Anoop-
>> >
>> > On Tue, Feb 2, 2016 at 12:07 PM, =E5=BC=A0=E9=93=8E <palomino219@gmail=
.com> wrote:
>> >
>> > > https://groups.google.com/forum/#!topic/protobuf/wAqvtPLBsE8
>> > >
>> > > PB2 and PB3 are wire compatible, and of course, protobuf-java is not
>> > > compatible so dependency will be a problem... But I think the shaded
>> > client
>> > > and server can solve the problem?
>> > >
>> > > Thanks.
>> > >
>> > > 2016-02-02 14:27 GMT+08:00 Stack <stack@duboce.net>:
>> > >
>> > > > We are running into a few issues with protobufs.
>> > > >
>> > > > + PB always copies all data before making a Message. This generate=
s
>> > > garbage
>> > > > unnecessarily.
>> > > > + CodedInputStream does not support ByteBuffers in 2.5. In 2.6 it
>> does
>> > > but
>> > > > again, copies the data out of the BB always; this is especially
>> painful
>> > > > when the BB is a DBB with its data offheap and intent is to keep
>> data
>> > > > offheap.
>> > > >
>> > > > There are other issues. CIS allocates 4k buffers regardless (See
>> > > > HBASE-15177).
>> > > > And then there was the HBaseZeroCopyByteString fun and games we ha=
d
>> a
>> > > while
>> > > > back.
>> > > >
>> > > > 3.0 PB adds UnsafeByteStrings so can do zero copy. Thats good. But
>> PB3
>> > is
>> > > > incompatible with PB2 (or at least, it looks like PB2 clients can'=
t
>> > talk
>> > > to
>> > > > PB3 [1]).
>> > > >
>> > > > There is javanano protobufs. All is open access, but it too looks
>> > > different
>> > > > to PB2 (i've not tried it).
>> > > >
>> > > > Protostuff seems really quiet these times [2].
>> > > >
>> > > > Fork (and shade)?
>> > > >
>> > > > Thoughts?
>> > > >
>> > > > St.Ack
>> > > >
>> > > > 1. https://github.com/google/protobuf/releases
>> > > > 2. https://groups.google.com/forum/#!forum/protostuff
>> > > >
>> > >
>> >
>>
>
>

--047d7b6725dc6447c3052acec64a--