hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Elliott Clark <ecl...@apache.org>
Subject Re: [DISCUSS] Shade protobuf so we can move to a newer version
Date Tue, 12 Apr 2016 23:53:24 GMT
Yeah I think it's a good idea however we need to remove PB from all our
user facing api's first.
There are some brave souls working on it here:
https://issues.apache.org/jira/browse/HBASE-15174

On Tue, Apr 12, 2016 at 4:17 PM, Stack <stack@duboce.net> wrote:

> I opened HBASE-15638 Shade protobuf but should probably raise the larger
> intent here. You fellows might have an opinion (smile).
>
> We need to shade PB so we can move to a different version.
>
> We need to move to a different version because we want to save on buffer
> copies -- newer versions of PB have some 'unsafe' facility so we can save
> on copies before serializing -- but we also want to be able to keep our
> data off heap always. Currently, our PB version expects byte arrays
> (2.5.0). Later versions of PBs have some support for ByteBuffer handling
> but even then, the BBs are expected to be onheap so we can't pass
> DirectByteBuffers (TODO: add support to PB to work w/ DBBs -- and BBs w/o
> copy -- and push these changes upstream).
>
> On an initial pass, the only difficult part seems to be interaction with
> HDFS in asyncwal (might just pull in the HDFS messages).
>
> The plan was to remove all references to protobuf and just have all modules
> depend on our hbase-protocol module. We'd bundle our PB in
> hbase-protocol.*.jar with packages offset to be at
> org.apache.hbase.shaded.com.google.protobuf. This way, our shading mess is
> contained some.
>
> Suggestions, opinions, all welcome.
>
> This project is part of the Ram and Anoop offheaping of the readpath and
> generally saving allocations effort.
>
> St.Ack
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message