hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject [DISCUSS] Shade protobuf so we can move to a newer version
Date Tue, 12 Apr 2016 23:17:30 GMT
I opened HBASE-15638 Shade protobuf but should probably raise the larger
intent here. You fellows might have an opinion (smile).

We need to shade PB so we can move to a different version.

We need to move to a different version because we want to save on buffer
copies -- newer versions of PB have some 'unsafe' facility so we can save
on copies before serializing -- but we also want to be able to keep our
data off heap always. Currently, our PB version expects byte arrays
(2.5.0). Later versions of PBs have some support for ByteBuffer handling
but even then, the BBs are expected to be onheap so we can't pass
DirectByteBuffers (TODO: add support to PB to work w/ DBBs -- and BBs w/o
copy -- and push these changes upstream).

On an initial pass, the only difficult part seems to be interaction with
HDFS in asyncwal (might just pull in the HDFS messages).

The plan was to remove all references to protobuf and just have all modules
depend on our hbase-protocol module. We'd bundle our PB in
hbase-protocol.*.jar with packages offset to be at
org.apache.hbase.shaded.com.google.protobuf. This way, our shading mess is
contained some.

Suggestions, opinions, all welcome.

This project is part of the Ram and Anoop offheaping of the readpath and
generally saving allocations effort.


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message