hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ramkrishna vasudevan <ramkrishna.s.vasude...@gmail.com>
Subject Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)
Date Mon, 03 Oct 2016 05:16:06 GMT
+1 on merge. Have been following up the JIRA from Stack and have discussed
on the upsides and downsides. +1 to do the merge.
Wire compatibility should be fine as per our testing. But still +1 to test
more in case we don't miss any case.

Regards
Ram

On Sun, Oct 2, 2016 at 11:41 PM, Andrew Purtell <andrew.purtell@gmail.com>
wrote:

> I don't know why you are responding to my email with this, but if is
> confusion as to how to install multiple versions of protoc safely, let me
> point you to the the '--prefix=' option of the ./configure script.
>
> <check out 2.5>
> ./configure --prefix=/opt/protobuf-2.5 && make && make install
>
> <check out 3.0>
> ./configure --prefix=/opt/protobuf-3.0 && make && make install
>
>
> > On Oct 2, 2016, at 10:41 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> >
> > I just installed protobuf 3 on my Mac.
> >
> > In case anyone doesn't have protobuf-2.5.0 source locally, please
> remember
> > to backup protoc executable before running 'sudo make install' under the
> > protobuf 3 dir.
> >
> > Cheers
> >
> > On Sun, Oct 2, 2016 at 10:18 AM, Andrew Purtell <
> andrew.purtell@gmail.com>
> > wrote:
> >
> >> I have 2.5 and 3.0 installed as /opt/protobuf-<version>, and have bash
> >> scripts that add the appropriate version's bin directory to the path.
> Not
> >> particularly onerous as I also have to switch between required JDK
> >> versions, so the scripts also set JAVA_HOME at and add JDK bin to the
> path
> >> for the required JDK for the build.
> >>
> >> Unlike with the scala compiler, which is after all JVM bytecode at a
> >> fundamental level, I don't think maven automation for automatic download
> >> and execution is possible. protoc is a native binary.
> >>
> >>> On Oct 1, 2016, at 11:30 PM, 张铎 <palomino219@gmail.com> wrote:
> >>>
> >>> Do we need to install protoc 3.0 manully before building HBase or the
> >> maven
> >>> protobuf plugin will automatically download the protoc compiler? Maybe
> we
> >>> need to install protoc 3.0 in the precommit docker file.
> >>>
> >>> 2016-10-02 14:20 GMT+08:00 张铎 <palomino219@gmail.com>:
> >>>
> >>>>
> >>>> 2016-10-02 0:50 GMT+08:00 Stack <stack@duboce.net>:
> >>>>
> >>>>>> On Sat, Oct 1, 2016 at 7:20 AM, 张铎 <palomino219@gmail.com>
wrote:
> >>>>>>
> >>>>>> Can we setup a compatibility checker job in jenkins? Start a
> >>>>> minicluster in
> >>>>>> one process, and use a client in another process to communicate
with
> >> it.
> >>>>>> The version of the client should be >= 0.98 and <= the
version of
> the
> >>>>>> minicluster. Of course we need to design the testing code carefully
> to
> >>>>> make
> >>>>>> sure that we have tested all the cases.
> >>>>>>
> >>>>>>
> >>>>> +1. We need this up and running before we put out an hbase-2.0.0.
I
> >> know
> >>>>> Matteo does this test manually on a regular basis but a formalization
> >>>>> would
> >>>>> help. I can add an exercise of Coprocessor Endpoints. I believe
this
> >> is on
> >>>>> Dima's list of TODOs but will let him speak for himself.
> >>>>>
> >>>>>
> >>>>>> And also I think we should make sure that no proto3 only feature
is
> >>>>>> introduced in our proto files until branch-1 is dead. Maybe
a
> >> precommit
> >>>>>> check?
> >>>>>>
> >>>>>>
> >>>>> I think you mean wire-format breaking changes?  Agree. We have our
> PB3
> >> set
> >>>>> to 2.5 compat mode and yes, we can't move on from this until we
are
> in
> >> a
> >>>>> place where we can say no to 2.5 clients.
> >>>>>
> >>>> Yes, for example, pb2.5 does not support map so we should not use map
> in
> >>>> our proto files.
> >>>>
> >>>>>
> >>>>> Making use of PB3isms cannot be avoided. PB3.1 adds a native
> >> replacement
> >>>>> for our HBaseZeroCopyByteString/ByteStringer hack. It also adds
> >> 'unsafe'
> >>>>> methods that we need to exploit if we are to keep our read/write
> paths
> >>>>> offheap.
> >>>>>
> >>>>> St.Ack
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>> Thanks.
> >>>>>>
> >>>>>> 2016-10-01 11:55 GMT+08:00 Sean Busbey <busbey@cloudera.com>:
> >>>>>>
> >>>>>>> have we experimentally confirmed that wire compatibility
is
> >>>>>>> maintained? I saw one mention of expecting wire compatibility
to be
> >>>>>>> fine, but nothing with someone using e.g. the clusterdock
work or
> >>>>>>> something to mix servers / clients or do replication.
> >>>>>>>
> >>>>>>>> On Fri, Sep 30, 2016 at 6:30 PM, Stack <stack@duboce.net>
wrote:
> >>>>>>>> I intend to do a mass commit late this weekend that
moves us on
> to a
> >>>>>>> shaded
> >>>>>>>> protobuf-3.1.0, either Sunday night or Monday morning.
> >>>>>>>>
> >>>>>>>> If objection, please speak up or if need more time for
> >>>>>>>> consideration/review, just shout.
> >>>>>>>>
> >>>>>>>> I want to merge the branch HBASE-16264 into master (it
is running
> >>>>> here
> >>>>>> up
> >>>>>>>> on jenkins https://builds.apache.org/
> view/H-L/view/HBase/job/HBASE-
> >>>>>>> 16264/).
> >>>>>>>> The branch at HBASE-16264 has three significant bodies-of-work
> that
> >>>>>>>> unfortunately are tangled and can only go in of a piece.
> >>>>>>>>
> >>>>>>>> * HBASE-16264 <https://issues.apache.org/jira/browse/HBASE-16264>
> >>>>> The
> >>>>>>>> shading of our protobuf usage so we can upgrade and/or
run with a
> >>>>>> patched
> >>>>>>>> protobuf WITHOUT breaking REST, Spark, and in particular,
> >>>>> Coprocessor
> >>>>>>>> Endpoints.
> >>>>>>>> * HBASE-16567 <https://issues.apache.org/jira/browse/HBASE-16567>
> >>>>> A
> >>>>>>> move
> >>>>>>>> up on to (shaded) protobuf-3.1.0
> >>>>>>>> * HBASE-16741 <https://issues.apache.org/jira/browse/HBASE-16741>
> >>>>> An
> >>>>>>>> amendment of our generate protobufs step to include
shading and a
> >>>>>>> bundling
> >>>>>>>> of protobuf src (with a means of calling a patch srcs
hook)
> >>>>>>>>
> >>>>>>>> Together we're talking about 40MB of change mostly made
of the
> >>>>> movement
> >>>>>>> of
> >>>>>>>> generated files or the application of a pattern that
alters where
> we
> >>>>>> get
> >>>>>>>> imports from. When done, you should notice no difference
and
> should
> >>>>> be
> >>>>>>> able
> >>>>>>>> to go about your business as per usual. Upside is that
we will be
> >>>>> able
> >>>>>> to
> >>>>>>>> avoid coming onheap doing protobuf marshalling/unmarshalling
as
> >>>>>> protobuf
> >>>>>>>> 2.5.0 requires. Downside is that we repeat a good portion
of our
> >>>>>> internal
> >>>>>>>> protos, once non-shaded so Coprocessor Endpoints can
keep working
> >>>>> and
> >>>>>>> then
> >>>>>>>> again as shaded for internal use.
> >>>>>>>>
> >>>>>>>> I provide some more overview below on the changes. See
the shading
> >>>>> doc
> >>>>>>>> here:
> >>>>>>>> https://docs.google.com/document/d/1H4NgLXQ9Y9KejwobddCqaVMEDCGby
> >>>>>>> DcXtdF5iAfDIEk/edit#
> >>>>>>>> for more detail (Patches are up on review board -- except
the
> latest
> >>>>>>>> HBASE-16264 which is too big for JIRA and RB). I am
currently
> >>>>> working
> >>>>>> on
> >>>>>>> a
> >>>>>>>> devs chapter for the book on protobuf going forward
that will go
> in
> >>>>> as
> >>>>>>> part
> >>>>>>>> of this patch.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> St.Ack
> >>>>>>>>
> >>>>>>>> Items of note:
> >>>>>>>>
> >>>>>>>> * Two new modules; one named hbase-protocol-shaded that
is used by
> >>>>>> hbase
> >>>>>>>> core. It has in it a shaded (and later patched) protobuf.
The
> other
> >>>>> new
> >>>>>>>> module is hbase-endpoint which goes after hbase-server
and has
> those
> >>>>>>>> bundled endpoints that I was able to break out of core
(there are
> a
> >>>>> few
> >>>>>>>> that are hopelessly entangled that need to be undone
as CPEPs but
> >>>>>>>> fortunately belong in core: Auth, Access, MultiRow).
> >>>>>>>> * I've tested running a branch-1 CPEP against a master
with these
> >>>>>>> patches
> >>>>>>>> in place and stuff like ACL (A CPEP) run from the branch-1
shell
> >>>>> work
> >>>>>>>> against the branch-2 server.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> On Mon, Aug 22, 2016 at 5:20 PM, Stack <stack@duboce.net>
wrote:
> >>>>>>>>>
> >>>>>>>>> This project goes on. I updated HBASE-1563 "Shade
protobuf" with
> >>>>> some
> >>>>>>> doc
> >>>>>>>>> on a final approach. We need to be able to refer
to both shaded
> and
> >>>>>>>>> non-shaded protobuf so we can support sending HDFS
old-school pb
> >>>>>>> Messages
> >>>>>>>>> but also so Coprocessor Endpoints keep working though
internally
> >>>>>>> protobufs
> >>>>>>>>> have been relocated. Funny you should ask, but yes,
there are
> some
> >>>>>>>>> downsides (as predicted by contributors on the JIRA).
I'd be
> >>>>>> interested
> >>>>>>> to
> >>>>>>>>> hear if they are too burdensome. In particular,
your IDE
> experience
> >>>>>>> gets a
> >>>>>>>>> little convoluted as you will need to add to your
build path, a
> jar
> >>>>>> with
> >>>>>>>>> the relocated pbs. A pain.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> St.Ack
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> On Wed, Apr 13, 2016 at 6:09 AM, Stack <stack@duboce.net>
> wrote:
> >>>>>>>>>>
> >>>>>>>>>> On Tue, Apr 12, 2016 at 9:26 PM, Sean Busbey
<busbey@apache.org
> >
> >>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>>> On Tue, Apr 12, 2016 at 6:17 PM, Stack
<stack@duboce.net>
> >> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On an initial pass, the only difficult
part seems to be
> >>>>>> interaction
> >>>>>>>>>>> with
> >>>>>>>>>>>> HDFS in asyncwal (might just pull in
the HDFS messages).
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> I have some idea how we can make this work
either by pushing
> >>>>>> asyncwal
> >>>>>>>>>>> upstream to HDFS or through some maven tricks,
depending on how
> >>>>> much
> >>>>>>>>>>> time we have.
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Maven tricks? Tell us more. Here or drop a note
up in the issue.
> >>>>>>>>>> Thanks Sean,
> >>>>>>>>>> St.Ack
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> busbey
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message