hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-15638) Shade protobuf
Date Tue, 04 Oct 2016 05:34:20 GMT

     [ https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

stack updated HBASE-15638:
       Resolution: Fixed
    Fix Version/s: 2.0.0
     Release Note: 
Shade/relocate and include the protobuf we use internally. See protobuf chapter in the refguide
for more on how we protobuf in hbase-.2.0.0 and going forward.

See https://docs.google.com/document/d/1H4NgLXQ9Y9KejwobddCqaVMEDCGbyDcXtdF5iAfDIEk/edit#
for how we arrived at this approach.

See http://mail-archives.apache.org/mod_mbox/hbase-dev/201610.mbox/%3C07850EDD-7230-431B-9AB0-C5C91B105EEC%40gmail.com%3E
for discussion around merging this change and of how we might revert if an alternative to
this awkward patch presents itself; e.g. an hadoop with CLASSPATH isolation (and means of
dealing with Sparks use of protobuf 2.5.0, etc.)
           Status: Resolved  (was: Patch Available)

Above set of patches adds shading of protobufs (and upgrades our internal protobuf to be 3.1.x).
Resolving. See release note for pointers on how we got to this awkward patch and of how we
would unto it if circumstances change.

> Shade protobuf
> --------------
>                 Key: HBASE-15638
>                 URL: https://issues.apache.org/jira/browse/HBASE-15638
>             Project: HBase
>          Issue Type: Bug
>          Components: Protobufs
>            Reporter: stack
>            Assignee: stack
>            Priority: Critical
>             Fix For: 2.0.0
>         Attachments: 15638v2.patch, HBASE-15638.master.001.patch, HBASE-15638.master.002.patch,
HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, HBASE-15638.master.003
(1).patch, HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, HBASE-15638.master.004.patch,
HBASE-15638.master.005.patch, HBASE-15638.master.006.patch, HBASE-15638.master.007.patch,
HBASE-15638.master.007.patch, HBASE-15638.master.008.patch, HBASE-15638.master.009.patch,
> We need to change our protobuf. Currently it is pb2.5.0. As is, protobufs expect all
buffers to be on-heap byte arrays. It does not have facility for dealing in ByteBuffers and
off-heap ByteBuffers in particular. This fact frustrates the off-heaping-of-the-write-path
project as marshalling/unmarshalling of protobufs involves a copy on-heap first.
> So, we need to patch our protobuf so it supports off-heap ByteBuffers. To ensure we pick
up the patched protobuf always, we need to relocate/shade our protobuf and adjust all protobuf
references accordingly.
> Given as we have protobufs in our public facing API, Coprocessor Endpoints -- which use
protobuf Service to describe new API -- a blind relocation/shading of com.google.protobuf.*
will break our API for CoProcessor EndPoints (CPEP) in particular. For example, in the Table
Interface, to invoke a method on a registered CPEP, we have:
> {code}<T extends com.google.protobuf.Service,R> Map<byte[],R> coprocessorService(
> Class<T> service, byte[] startKey, byte[] endKey,                             
               org.apache.hadoop.hbase.client.coprocessor.Batch.Call<T,R> callable)
> throws com.google.protobuf.ServiceException, Throwable{code}
> This issue is how we intend to shade protobuf for hbase-2.0.0 while preserving our API
as is so CPEPs continue to work on the new hbase.

This message was sent by Atlassian JIRA

View raw message