hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-15638) Shade protobuf
Date Wed, 07 Sep 2016 03:51:20 GMT

     [ https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

stack updated HBASE-15638:
--------------------------
    Description: 
We need to change our protobuf. Currently it is pb2.5.0. As is, protobufs expect all buffers
to be on-heap byte arrays. It does not have facility for dealing in ByteBuffers and off-heap
ByteBuffers in particular. This fact frustrates the off-heaping-of-the-write-path project
as marshalling/unmarshalling of protobufs involves a copy on-heap first.

So, we need to patch our protobuf so it supports off-heap ByteBuffers. To ensure we pick up
the patched protobuf always, we need to relocate/shade our protobuf and adjust all protobuf
references accordingly.

Given as we have protobufs in our public facing API, Coprocessor Endpoints -- which use protobuf
Service to describe new API -- a blind relocation/shading of com.google.protobuf.* will break
our API for CoProcessor EndPoints (CPEP) in particular. For example, in the Table Interface,
to invoke a method on a registered CPEP, we have:

<T extends com.google.protobuf.Service,R> Map<byte[],R> coprocessorService(
Class<T> service, byte[] startKey, byte[] endKey,                                  
          org.apache.hadoop.hbase.client.coprocessor.Batch.Call<T,R> callable)
throws com.google.protobuf.ServiceException, Throwable

This issue is how we intend to shade protobuf for hbase-2.0.0 while preserving our API as
is so CPEPs continue to work on the new hbase.

  was:
Shade protobufs so we can move to a different version without breaking the world. We want
to get up on pb3 because it has unsafe methods that allow us save on copies; it also has some
means of dealing with BBs so we can pass it offheap DBBs. We'll probably want to change PB3
to open it up some more too so we can stay offheap as we traverse PB. This issue comes of
[~anoop.hbase] and [~ram_krish]'s offheaping of the readpath work.

This change is mostly straight-forward but there are some tricky bits:

 # How to interface with HDFS? It wants its ByteStrings. Here in particular in FanOutOneBlockAsyncDFSOutputSaslHelper:

{code}
      if (payload != null) {
        builder.setPayload(ByteString.copyFrom(payload));
      }
{code}

 # [~busbey] also points out that we need to take care of endpoints done as pb. Test at least.

Let me raise this one on the dev list too.



> Shade protobuf
> --------------
>
>                 Key: HBASE-15638
>                 URL: https://issues.apache.org/jira/browse/HBASE-15638
>             Project: HBase
>          Issue Type: Bug
>          Components: Protobufs
>            Reporter: stack
>            Assignee: stack
>            Priority: Critical
>         Attachments: 15638v2.patch, HBASE-15638.master.001.patch, HBASE-15638.master.002.patch,
HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, HBASE-15638.master.003
(1).patch, HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, HBASE-15638.master.004.patch,
HBASE-15638.master.005.patch, HBASE-15638.master.006.patch, HBASE-15638.master.007.patch,
HBASE-15638.master.007.patch, HBASE-15638.master.008.patch, HBASE-15638.master.009.patch,
as.far.as.server.patch
>
>
> We need to change our protobuf. Currently it is pb2.5.0. As is, protobufs expect all
buffers to be on-heap byte arrays. It does not have facility for dealing in ByteBuffers and
off-heap ByteBuffers in particular. This fact frustrates the off-heaping-of-the-write-path
project as marshalling/unmarshalling of protobufs involves a copy on-heap first.
> So, we need to patch our protobuf so it supports off-heap ByteBuffers. To ensure we pick
up the patched protobuf always, we need to relocate/shade our protobuf and adjust all protobuf
references accordingly.
> Given as we have protobufs in our public facing API, Coprocessor Endpoints -- which use
protobuf Service to describe new API -- a blind relocation/shading of com.google.protobuf.*
will break our API for CoProcessor EndPoints (CPEP) in particular. For example, in the Table
Interface, to invoke a method on a registered CPEP, we have:
> <T extends com.google.protobuf.Service,R> Map<byte[],R> coprocessorService(
> Class<T> service, byte[] startKey, byte[] endKey,                             
               org.apache.hadoop.hbase.client.coprocessor.Batch.Call<T,R> callable)
> throws com.google.protobuf.ServiceException, Throwable
> This issue is how we intend to shade protobuf for hbase-2.0.0 while preserving our API
as is so CPEPs continue to work on the new hbase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message