hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)
Date Mon, 03 Oct 2016 16:32:34 GMT
On Mon, Oct 3, 2016 at 7:16 AM, Ted Yu <yuzhihong@gmail.com> wrote:

> Looks like compile-protobuf profile is not in hbase-protocol-shaded/pom.xml
> (in HBASE-16264 branch)
>
>
Sorry. I don't get what you are saying here

(The target in the new module is generate-sources. See the included README.
This step does more work now more than just generating protocs, hence new
profile name.)



> Seems precommit jobs should pass with the current formation.
>
>
Are you stating that this patch is likely to build? (Yes, the patch I
submitted builds).



> In the future, if we add another profile for compiling proto3 files, we
> need to specify the path to proto3 compiler.
>
>

> Please correct me if I am wrong.
>
>
I don't know what you are asking. Why do we have to specify 'paths'? We
don't have to currently (See the plugin we use generating protos now from
hadoop). Maybe you are trying to distinguish the production of protobuf2.5
vs 3.1 protos but these are isolated by module....


I said I'd commit this morning but let me wait a while. There may be some
more questions/objections and I'd like to have a clean build up on jenkins
here [1] before I commit (jenkins is being ornery).

St.Ack
1.
https://builds.apache.org/job/HBASE-16264/jdk=JDK%201.8%20(latest),label=yahoo-not-h2/28/console

Service Unavailable

The server is temporarily unable to service your
request due to maintenance downtime or capacity
problems. Please try again later.






> On Mon, Oct 3, 2016 at 6:58 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>
> > The protoc generated files (such as MasterProtos) are checked into source
> > repo, right ?
> >
> > Do we need proto3 on the precommit image(s) ?
> >
> > On Mon, Oct 3, 2016 at 5:18 AM, 张铎 <palomino219@gmail.com> wrote:
> >
> >> Then I think we need to file an issue to change the protoc version
> >> installed in the precommit docker file to 3.x before the merge.
> Otherwise
> >> the precommit build for protoc check maybe broken after the merge...
> >>
> >>
> >> 2016-10-03 1:18 GMT+08:00 Andrew Purtell <andrew.purtell@gmail.com>:
> >>
> >> > I have 2.5 and 3.0 installed as /opt/protobuf-<version>, and have
bash
> >> > scripts that add the appropriate version's bin directory to the path.
> >> Not
> >> > particularly onerous as I also have to switch between required JDK
> >> > versions, so the scripts also set JAVA_HOME at and add JDK bin to the
> >> path
> >> > for the required JDK for the build.
> >> >
> >> > Unlike with the scala compiler, which is after all JVM bytecode at a
> >> > fundamental level, I don't think maven automation for automatic
> download
> >> > and execution is possible. protoc is a native binary.
> >> >
> >> > > On Oct 1, 2016, at 11:30 PM, 张铎 <palomino219@gmail.com>
wrote:
> >> > >
> >> > > Do we need to install protoc 3.0 manully before building HBase or
> the
> >> > maven
> >> > > protobuf plugin will automatically download the protoc compiler?
> >> Maybe we
> >> > > need to install protoc 3.0 in the precommit docker file.
> >> > >
> >> > > 2016-10-02 14:20 GMT+08:00 张铎 <palomino219@gmail.com>:
> >> > >
> >> > >>
> >> > >> 2016-10-02 0:50 GMT+08:00 Stack <stack@duboce.net>:
> >> > >>
> >> > >>>> On Sat, Oct 1, 2016 at 7:20 AM, 张铎 <palomino219@gmail.com>
> wrote:
> >> > >>>>
> >> > >>>> Can we setup a compatibility checker job in jenkins? Start
a
> >> > >>> minicluster in
> >> > >>>> one process, and use a client in another process to communicate
> >> with
> >> > it.
> >> > >>>> The version of the client should be >= 0.98 and <=
the version of
> >> the
> >> > >>>> minicluster. Of course we need to design the testing code
> >> carefully to
> >> > >>> make
> >> > >>>> sure that we have tested all the cases.
> >> > >>>>
> >> > >>>>
> >> > >>> +1. We need this up and running before we put out an hbase-2.0.0.
> I
> >> > know
> >> > >>> Matteo does this test manually on a regular basis but a
> >> formalization
> >> > >>> would
> >> > >>> help. I can add an exercise of Coprocessor Endpoints. I believe
> this
> >> > is on
> >> > >>> Dima's list of TODOs but will let him speak for himself.
> >> > >>>
> >> > >>>
> >> > >>>> And also I think we should make sure that no proto3 only
feature
> is
> >> > >>>> introduced in our proto files until branch-1 is dead.
Maybe a
> >> > precommit
> >> > >>>> check?
> >> > >>>>
> >> > >>>>
> >> > >>> I think you mean wire-format breaking changes?  Agree. We
have our
> >> PB3
> >> > set
> >> > >>> to 2.5 compat mode and yes, we can't move on from this until
we
> are
> >> in
> >> > a
> >> > >>> place where we can say no to 2.5 clients.
> >> > >>>
> >> > >> Yes, for example, pb2.5 does not support map so we should not
use
> >> map in
> >> > >> our proto files.
> >> > >>
> >> > >>>
> >> > >>> Making use of PB3isms cannot be avoided. PB3.1 adds a native
> >> > replacement
> >> > >>> for our HBaseZeroCopyByteString/ByteStringer hack. It also
adds
> >> > 'unsafe'
> >> > >>> methods that we need to exploit if we are to keep our read/write
> >> paths
> >> > >>> offheap.
> >> > >>>
> >> > >>> St.Ack
> >> > >>>
> >> > >>>
> >> > >>>
> >> > >>>
> >> > >>>
> >> > >>>> Thanks.
> >> > >>>>
> >> > >>>> 2016-10-01 11:55 GMT+08:00 Sean Busbey <busbey@cloudera.com>:
> >> > >>>>
> >> > >>>>> have we experimentally confirmed that wire compatibility
is
> >> > >>>>> maintained? I saw one mention of expecting wire compatibility
to
> >> be
> >> > >>>>> fine, but nothing with someone using e.g. the clusterdock
work
> or
> >> > >>>>> something to mix servers / clients or do replication.
> >> > >>>>>
> >> > >>>>>> On Fri, Sep 30, 2016 at 6:30 PM, Stack <stack@duboce.net>
> wrote:
> >> > >>>>>> I intend to do a mass commit late this weekend
that moves us on
> >> to a
> >> > >>>>> shaded
> >> > >>>>>> protobuf-3.1.0, either Sunday night or Monday
morning.
> >> > >>>>>>
> >> > >>>>>> If objection, please speak up or if need more
time for
> >> > >>>>>> consideration/review, just shout.
> >> > >>>>>>
> >> > >>>>>> I want to merge the branch HBASE-16264 into master
(it is
> running
> >> > >>> here
> >> > >>>> up
> >> > >>>>>> on jenkins https://builds.apache.org/view
> >> /H-L/view/HBase/job/HBASE-
> >> > >>>>> 16264/).
> >> > >>>>>> The branch at HBASE-16264 has three significant
bodies-of-work
> >> that
> >> > >>>>>> unfortunately are tangled and can only go in of
a piece.
> >> > >>>>>>
> >> > >>>>>> * HBASE-16264 <https://issues.apache.org/
> jira/browse/HBASE-16264
> >> >
> >> > >>> The
> >> > >>>>>> shading of our protobuf usage so we can upgrade
and/or run
> with a
> >> > >>>> patched
> >> > >>>>>> protobuf WITHOUT breaking REST, Spark, and in
particular,
> >> > >>> Coprocessor
> >> > >>>>>> Endpoints.
> >> > >>>>>> * HBASE-16567 <https://issues.apache.org/
> jira/browse/HBASE-16567
> >> >
> >> > >>> A
> >> > >>>>> move
> >> > >>>>>> up on to (shaded) protobuf-3.1.0
> >> > >>>>>> * HBASE-16741 <https://issues.apache.org/
> jira/browse/HBASE-16741
> >> >
> >> > >>> An
> >> > >>>>>> amendment of our generate protobufs step to include
shading
> and a
> >> > >>>>> bundling
> >> > >>>>>> of protobuf src (with a means of calling a patch
srcs hook)
> >> > >>>>>>
> >> > >>>>>> Together we're talking about 40MB of change mostly
made of the
> >> > >>> movement
> >> > >>>>> of
> >> > >>>>>> generated files or the application of a pattern
that alters
> >> where we
> >> > >>>> get
> >> > >>>>>> imports from. When done, you should notice no
difference and
> >> should
> >> > >>> be
> >> > >>>>> able
> >> > >>>>>> to go about your business as per usual. Upside
is that we will
> be
> >> > >>> able
> >> > >>>> to
> >> > >>>>>> avoid coming onheap doing protobuf marshalling/unmarshalling
as
> >> > >>>> protobuf
> >> > >>>>>> 2.5.0 requires. Downside is that we repeat a good
portion of
> our
> >> > >>>> internal
> >> > >>>>>> protos, once non-shaded so Coprocessor Endpoints
can keep
> working
> >> > >>> and
> >> > >>>>> then
> >> > >>>>>> again as shaded for internal use.
> >> > >>>>>>
> >> > >>>>>> I provide some more overview below on the changes.
See the
> >> shading
> >> > >>> doc
> >> > >>>>>> here:
> >> > >>>>>> https://docs.google.com/document/d/1H4NgLXQ9Y9KejwobddCqaVME
> >> DCGby
> >> > >>>>> DcXtdF5iAfDIEk/edit#
> >> > >>>>>> for more detail (Patches are up on review board
-- except the
> >> latest
> >> > >>>>>> HBASE-16264 which is too big for JIRA and RB).
I am currently
> >> > >>> working
> >> > >>>> on
> >> > >>>>> a
> >> > >>>>>> devs chapter for the book on protobuf going forward
that will
> go
> >> in
> >> > >>> as
> >> > >>>>> part
> >> > >>>>>> of this patch.
> >> > >>>>>>
> >> > >>>>>> Thanks,
> >> > >>>>>> St.Ack
> >> > >>>>>>
> >> > >>>>>> Items of note:
> >> > >>>>>>
> >> > >>>>>> * Two new modules; one named hbase-protocol-shaded
that is used
> >> by
> >> > >>>> hbase
> >> > >>>>>> core. It has in it a shaded (and later patched)
protobuf. The
> >> other
> >> > >>> new
> >> > >>>>>> module is hbase-endpoint which goes after hbase-server
and has
> >> those
> >> > >>>>>> bundled endpoints that I was able to break out
of core (there
> >> are a
> >> > >>> few
> >> > >>>>>> that are hopelessly entangled that need to be
undone as CPEPs
> but
> >> > >>>>>> fortunately belong in core: Auth, Access, MultiRow).
> >> > >>>>>> * I've tested running a branch-1 CPEP against
a master with
> these
> >> > >>>>> patches
> >> > >>>>>> in place and stuff like ACL (A CPEP) run from
the branch-1
> shell
> >> > >>> work
> >> > >>>>>> against the branch-2 server.
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>>> On Mon, Aug 22, 2016 at 5:20 PM, Stack <stack@duboce.net>
> >> wrote:
> >> > >>>>>>>
> >> > >>>>>>> This project goes on. I updated HBASE-1563
"Shade protobuf"
> with
> >> > >>> some
> >> > >>>>> doc
> >> > >>>>>>> on a final approach. We need to be able to
refer to both
> shaded
> >> and
> >> > >>>>>>> non-shaded protobuf so we can support sending
HDFS old-school
> pb
> >> > >>>>> Messages
> >> > >>>>>>> but also so Coprocessor Endpoints keep working
though
> internally
> >> > >>>>> protobufs
> >> > >>>>>>> have been relocated. Funny you should ask,
but yes, there are
> >> some
> >> > >>>>>>> downsides (as predicted by contributors on
the JIRA). I'd be
> >> > >>>> interested
> >> > >>>>> to
> >> > >>>>>>> hear if they are too burdensome. In particular,
your IDE
> >> experience
> >> > >>>>> gets a
> >> > >>>>>>> little convoluted as you will need to add
to your build path,
> a
> >> jar
> >> > >>>> with
> >> > >>>>>>> the relocated pbs. A pain.
> >> > >>>>>>>
> >> > >>>>>>> Thanks,
> >> > >>>>>>> St.Ack
> >> > >>>>>>>
> >> > >>>>>>>
> >> > >>>>>>>> On Wed, Apr 13, 2016 at 6:09 AM, Stack
<stack@duboce.net>
> >> wrote:
> >> > >>>>>>>>
> >> > >>>>>>>> On Tue, Apr 12, 2016 at 9:26 PM, Sean
Busbey <
> >> busbey@apache.org>
> >> > >>>>> wrote:
> >> > >>>>>>>>
> >> > >>>>>>>>>> On Tue, Apr 12, 2016 at 6:17 PM,
Stack <stack@duboce.net>
> >> > wrote:
> >> > >>>>>>>>>>
> >> > >>>>>>>>>>
> >> > >>>>>>>>>> On an initial pass, the only difficult
part seems to be
> >> > >>>> interaction
> >> > >>>>>>>>> with
> >> > >>>>>>>>>> HDFS in asyncwal (might just pull
in the HDFS messages).
> >> > >>>>>>>>>>
> >> > >>>>>>>>>>
> >> > >>>>>>>>>
> >> > >>>>>>>>> I have some idea how we can make this
work either by pushing
> >> > >>>> asyncwal
> >> > >>>>>>>>> upstream to HDFS or through some maven
tricks, depending on
> >> how
> >> > >>> much
> >> > >>>>>>>>> time we have.
> >> > >>>>>>>>>
> >> > >>>>>>>>
> >> > >>>>>>>> Maven tricks? Tell us more. Here or drop
a note up in the
> >> issue.
> >> > >>>>>>>> Thanks Sean,
> >> > >>>>>>>> St.Ack
> >> > >>>>>>>>
> >> > >>>>>>>
> >> > >>>>>>>
> >> > >>>>>
> >> > >>>>>
> >> > >>>>>
> >> > >>>>> --
> >> > >>>>> busbey
> >> > >>>>>
> >> > >>>>
> >> > >>>
> >> > >>
> >> > >>
> >> >
> >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message