hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)
Date Fri, 30 Sep 2016 23:30:29 GMT
I intend to do a mass commit late this weekend that moves us on to a shaded
protobuf-3.1.0, either Sunday night or Monday morning.

If objection, please speak up or if need more time for
consideration/review, just shout.

I want to merge the branch HBASE-16264 into master (it is running here up
on jenkins https://builds.apache.org/view/H-L/view/HBase/job/HBASE-16264/).
The branch at HBASE-16264 has three significant bodies-of-work that
unfortunately are tangled and can only go in of a piece.

 * HBASE-16264 <https://issues.apache.org/jira/browse/HBASE-16264> The
shading of our protobuf usage so we can upgrade and/or run with a patched
protobuf WITHOUT breaking REST, Spark, and in particular, Coprocessor
 * HBASE-16567 <https://issues.apache.org/jira/browse/HBASE-16567> A move
up on to (shaded) protobuf-3.1.0
 * HBASE-16741 <https://issues.apache.org/jira/browse/HBASE-16741> An
amendment of our generate protobufs step to include shading and a bundling
of protobuf src (with a means of calling a patch srcs hook)

Together we're talking about 40MB of change mostly made of the movement of
generated files or the application of a pattern that alters where we get
imports from. When done, you should notice no difference and should be able
to go about your business as per usual. Upside is that we will be able to
avoid coming onheap doing protobuf marshalling/unmarshalling as protobuf
2.5.0 requires. Downside is that we repeat a good portion of our internal
protos, once non-shaded so Coprocessor Endpoints can keep working and then
again as shaded for internal use.

I provide some more overview below on the changes. See the shading doc
for more detail (Patches are up on review board -- except the latest
HBASE-16264 which is too big for JIRA and RB). I am currently working on a
devs chapter for the book on protobuf going forward that will go in as part
of this patch.


Items of note:

 * Two new modules; one named hbase-protocol-shaded that is used by hbase
core. It has in it a shaded (and later patched) protobuf. The other new
module is hbase-endpoint which goes after hbase-server and has those
bundled endpoints that I was able to break out of core (there are a few
that are hopelessly entangled that need to be undone as CPEPs but
fortunately belong in core: Auth, Access, MultiRow).
 * I've tested running a branch-1 CPEP against a master with these patches
in place and stuff like ACL (A CPEP) run from the branch-1 shell work
against the branch-2 server.

On Mon, Aug 22, 2016 at 5:20 PM, Stack <stack@duboce.net> wrote:

> This project goes on. I updated HBASE-1563 "Shade protobuf" with some doc
> on a final approach. We need to be able to refer to both shaded and
> non-shaded protobuf so we can support sending HDFS old-school pb Messages
> but also so Coprocessor Endpoints keep working though internally protobufs
> have been relocated. Funny you should ask, but yes, there are some
> downsides (as predicted by contributors on the JIRA). I'd be interested to
> hear if they are too burdensome. In particular, your IDE experience gets a
> little convoluted as you will need to add to your build path, a jar with
> the relocated pbs. A pain.
> Thanks,
> St.Ack
> On Wed, Apr 13, 2016 at 6:09 AM, Stack <stack@duboce.net> wrote:
>> On Tue, Apr 12, 2016 at 9:26 PM, Sean Busbey <busbey@apache.org> wrote:
>>> On Tue, Apr 12, 2016 at 6:17 PM, Stack <stack@duboce.net> wrote:
>>> >
>>> >
>>> > On an initial pass, the only difficult part seems to be interaction
>>> with
>>> > HDFS in asyncwal (might just pull in the HDFS messages).
>>> >
>>> >
>>> I have some idea how we can make this work either by pushing asyncwal
>>> upstream to HDFS or through some maven tricks, depending on how much
>>> time we have.
>> Maven tricks? Tell us more. Here or drop a note up in the issue.
>> Thanks Sean,
>> St.Ack

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message