hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-15638) Shade protobuf
Date Wed, 20 Jul 2016 00:20:20 GMT

    [ https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15385114#comment-15385114

stack commented on HBASE-15638:

Reassessing your suggestion @busbey:

bq. if all references to our internal use of protobuf are in the hbase-protocol module, then
we needn't have any references to the relocated packages, because we can have the shade plugin
take care of rewriting them just in that module while including the relocated classes that
we use within the jar.

We can't do this. pb exceptions are throughout the codebase (could clean these up) as is the
ByteString type and ipc is doing direct pb Type slinging. Would take a good bit of disruptive
work looking at the size of the last patch attached.

bq. If we go the route of a module that relocated protobuf, we could make a profile in our
project / top level pom "uses-relocated-protobuf". For those modules that activate it, we
can then use the shade plugin to rewrite the references from the original protobuf to the
shaded one. 

We could go this route but I think the profile on certain modules would be exotic and then
we have to move out the hdfs pb references anyways.

Would you have a few minutes to chat on the above just so I'm sure I am clear on what you
were suggesting? Thanks [~busbey]

> Shade protobuf
> --------------
>                 Key: HBASE-15638
>                 URL: https://issues.apache.org/jira/browse/HBASE-15638
>             Project: HBase
>          Issue Type: Bug
>          Components: Protobufs
>            Reporter: stack
>         Attachments: 15638v2.patch, as.far.as.server.patch
> Shade protobufs so we can move to a different version without breaking the world. We
want to get up on pb3 because it has unsafe methods that allow us save on copies; it also
has some means of dealing with BBs so we can pass it offheap DBBs. We'll probably want to
change PB3 to open it up some more too so we can stay offheap as we traverse PB. This issue
comes of [~anoop.hbase] and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>       if (payload != null) {
>         builder.setPayload(ByteString.copyFrom(payload));
>       }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as pb. Test
at least.
> Let me raise this one on the dev list too.

This message was sent by Atlassian JIRA

View raw message