hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 张铎 <palomino...@gmail.com>
Subject Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)
Date Mon, 03 Oct 2016 12:18:36 GMT
Then I think we need to file an issue to change the protoc version
installed in the precommit docker file to 3.x before the merge. Otherwise
the precommit build for protoc check maybe broken after the merge...

2016-10-03 1:18 GMT+08:00 Andrew Purtell <andrew.purtell@gmail.com>:

> I have 2.5 and 3.0 installed as /opt/protobuf-<version>, and have bash
> scripts that add the appropriate version's bin directory to the path. Not
> particularly onerous as I also have to switch between required JDK
> versions, so the scripts also set JAVA_HOME at and add JDK bin to the path
> for the required JDK for the build.
>
> Unlike with the scala compiler, which is after all JVM bytecode at a
> fundamental level, I don't think maven automation for automatic download
> and execution is possible. protoc is a native binary.
>
> > On Oct 1, 2016, at 11:30 PM, 张铎 <palomino219@gmail.com> wrote:
> >
> > Do we need to install protoc 3.0 manully before building HBase or the
> maven
> > protobuf plugin will automatically download the protoc compiler? Maybe we
> > need to install protoc 3.0 in the precommit docker file.
> >
> > 2016-10-02 14:20 GMT+08:00 张铎 <palomino219@gmail.com>:
> >
> >>
> >> 2016-10-02 0:50 GMT+08:00 Stack <stack@duboce.net>:
> >>
> >>>> On Sat, Oct 1, 2016 at 7:20 AM, 张铎 <palomino219@gmail.com>
wrote:
> >>>>
> >>>> Can we setup a compatibility checker job in jenkins? Start a
> >>> minicluster in
> >>>> one process, and use a client in another process to communicate with
> it.
> >>>> The version of the client should be >= 0.98 and <= the version
of the
> >>>> minicluster. Of course we need to design the testing code carefully
to
> >>> make
> >>>> sure that we have tested all the cases.
> >>>>
> >>>>
> >>> +1. We need this up and running before we put out an hbase-2.0.0. I
> know
> >>> Matteo does this test manually on a regular basis but a formalization
> >>> would
> >>> help. I can add an exercise of Coprocessor Endpoints. I believe this
> is on
> >>> Dima's list of TODOs but will let him speak for himself.
> >>>
> >>>
> >>>> And also I think we should make sure that no proto3 only feature is
> >>>> introduced in our proto files until branch-1 is dead. Maybe a
> precommit
> >>>> check?
> >>>>
> >>>>
> >>> I think you mean wire-format breaking changes?  Agree. We have our PB3
> set
> >>> to 2.5 compat mode and yes, we can't move on from this until we are in
> a
> >>> place where we can say no to 2.5 clients.
> >>>
> >> Yes, for example, pb2.5 does not support map so we should not use map in
> >> our proto files.
> >>
> >>>
> >>> Making use of PB3isms cannot be avoided. PB3.1 adds a native
> replacement
> >>> for our HBaseZeroCopyByteString/ByteStringer hack. It also adds
> 'unsafe'
> >>> methods that we need to exploit if we are to keep our read/write paths
> >>> offheap.
> >>>
> >>> St.Ack
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>> Thanks.
> >>>>
> >>>> 2016-10-01 11:55 GMT+08:00 Sean Busbey <busbey@cloudera.com>:
> >>>>
> >>>>> have we experimentally confirmed that wire compatibility is
> >>>>> maintained? I saw one mention of expecting wire compatibility to
be
> >>>>> fine, but nothing with someone using e.g. the clusterdock work or
> >>>>> something to mix servers / clients or do replication.
> >>>>>
> >>>>>> On Fri, Sep 30, 2016 at 6:30 PM, Stack <stack@duboce.net>
wrote:
> >>>>>> I intend to do a mass commit late this weekend that moves us
on to a
> >>>>> shaded
> >>>>>> protobuf-3.1.0, either Sunday night or Monday morning.
> >>>>>>
> >>>>>> If objection, please speak up or if need more time for
> >>>>>> consideration/review, just shout.
> >>>>>>
> >>>>>> I want to merge the branch HBASE-16264 into master (it is running
> >>> here
> >>>> up
> >>>>>> on jenkins https://builds.apache.org/view/H-L/view/HBase/job/HBASE-
> >>>>> 16264/).
> >>>>>> The branch at HBASE-16264 has three significant bodies-of-work
that
> >>>>>> unfortunately are tangled and can only go in of a piece.
> >>>>>>
> >>>>>> * HBASE-16264 <https://issues.apache.org/jira/browse/HBASE-16264>
> >>> The
> >>>>>> shading of our protobuf usage so we can upgrade and/or run with
a
> >>>> patched
> >>>>>> protobuf WITHOUT breaking REST, Spark, and in particular,
> >>> Coprocessor
> >>>>>> Endpoints.
> >>>>>> * HBASE-16567 <https://issues.apache.org/jira/browse/HBASE-16567>
> >>> A
> >>>>> move
> >>>>>> up on to (shaded) protobuf-3.1.0
> >>>>>> * HBASE-16741 <https://issues.apache.org/jira/browse/HBASE-16741>
> >>> An
> >>>>>> amendment of our generate protobufs step to include shading
and a
> >>>>> bundling
> >>>>>> of protobuf src (with a means of calling a patch srcs hook)
> >>>>>>
> >>>>>> Together we're talking about 40MB of change mostly made of the
> >>> movement
> >>>>> of
> >>>>>> generated files or the application of a pattern that alters
where we
> >>>> get
> >>>>>> imports from. When done, you should notice no difference and
should
> >>> be
> >>>>> able
> >>>>>> to go about your business as per usual. Upside is that we will
be
> >>> able
> >>>> to
> >>>>>> avoid coming onheap doing protobuf marshalling/unmarshalling
as
> >>>> protobuf
> >>>>>> 2.5.0 requires. Downside is that we repeat a good portion of
our
> >>>> internal
> >>>>>> protos, once non-shaded so Coprocessor Endpoints can keep working
> >>> and
> >>>>> then
> >>>>>> again as shaded for internal use.
> >>>>>>
> >>>>>> I provide some more overview below on the changes. See the shading
> >>> doc
> >>>>>> here:
> >>>>>> https://docs.google.com/document/d/1H4NgLXQ9Y9KejwobddCqaVMEDCGby
> >>>>> DcXtdF5iAfDIEk/edit#
> >>>>>> for more detail (Patches are up on review board -- except the
latest
> >>>>>> HBASE-16264 which is too big for JIRA and RB). I am currently
> >>> working
> >>>> on
> >>>>> a
> >>>>>> devs chapter for the book on protobuf going forward that will
go in
> >>> as
> >>>>> part
> >>>>>> of this patch.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> St.Ack
> >>>>>>
> >>>>>> Items of note:
> >>>>>>
> >>>>>> * Two new modules; one named hbase-protocol-shaded that is used
by
> >>>> hbase
> >>>>>> core. It has in it a shaded (and later patched) protobuf. The
other
> >>> new
> >>>>>> module is hbase-endpoint which goes after hbase-server and has
those
> >>>>>> bundled endpoints that I was able to break out of core (there
are a
> >>> few
> >>>>>> that are hopelessly entangled that need to be undone as CPEPs
but
> >>>>>> fortunately belong in core: Auth, Access, MultiRow).
> >>>>>> * I've tested running a branch-1 CPEP against a master with
these
> >>>>> patches
> >>>>>> in place and stuff like ACL (A CPEP) run from the branch-1 shell
> >>> work
> >>>>>> against the branch-2 server.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> On Mon, Aug 22, 2016 at 5:20 PM, Stack <stack@duboce.net>
wrote:
> >>>>>>>
> >>>>>>> This project goes on. I updated HBASE-1563 "Shade protobuf"
with
> >>> some
> >>>>> doc
> >>>>>>> on a final approach. We need to be able to refer to both
shaded and
> >>>>>>> non-shaded protobuf so we can support sending HDFS old-school
pb
> >>>>> Messages
> >>>>>>> but also so Coprocessor Endpoints keep working though internally
> >>>>> protobufs
> >>>>>>> have been relocated. Funny you should ask, but yes, there
are some
> >>>>>>> downsides (as predicted by contributors on the JIRA). I'd
be
> >>>> interested
> >>>>> to
> >>>>>>> hear if they are too burdensome. In particular, your IDE
experience
> >>>>> gets a
> >>>>>>> little convoluted as you will need to add to your build
path, a jar
> >>>> with
> >>>>>>> the relocated pbs. A pain.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> St.Ack
> >>>>>>>
> >>>>>>>
> >>>>>>>> On Wed, Apr 13, 2016 at 6:09 AM, Stack <stack@duboce.net>
wrote:
> >>>>>>>>
> >>>>>>>> On Tue, Apr 12, 2016 at 9:26 PM, Sean Busbey <busbey@apache.org>
> >>>>> wrote:
> >>>>>>>>
> >>>>>>>>>> On Tue, Apr 12, 2016 at 6:17 PM, Stack <stack@duboce.net>
> wrote:
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On an initial pass, the only difficult part
seems to be
> >>>> interaction
> >>>>>>>>> with
> >>>>>>>>>> HDFS in asyncwal (might just pull in the HDFS
messages).
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> I have some idea how we can make this work either
by pushing
> >>>> asyncwal
> >>>>>>>>> upstream to HDFS or through some maven tricks, depending
on how
> >>> much
> >>>>>>>>> time we have.
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> Maven tricks? Tell us more. Here or drop a note up in
the issue.
> >>>>>>>> Thanks Sean,
> >>>>>>>> St.Ack
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> busbey
> >>>>>
> >>>>
> >>>
> >>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message