hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anoop John <anoop.hb...@gmail.com>
Subject Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)
Date Mon, 03 Oct 2016 05:44:45 GMT
Yes we are not changing the proto files as such.. We use the old ones
but regenerate the java files using PB 3 compiler.  But within our
code we will be using PB 3 specific features/APIs . Eg: As Stack
mentioned, the way to wrap a byte[] into PB ByteString.

-Anoop-

On Mon, Oct 3, 2016 at 10:46 AM, ramkrishna vasudevan
<ramkrishna.s.vasudevan@gmail.com> wrote:
> +1 on merge. Have been following up the JIRA from Stack and have discussed
> on the upsides and downsides. +1 to do the merge.
> Wire compatibility should be fine as per our testing. But still +1 to test
> more in case we don't miss any case.
>
> Regards
> Ram
>
> On Sun, Oct 2, 2016 at 11:41 PM, Andrew Purtell <andrew.purtell@gmail.com>
> wrote:
>
>> I don't know why you are responding to my email with this, but if is
>> confusion as to how to install multiple versions of protoc safely, let me
>> point you to the the '--prefix=' option of the ./configure script.
>>
>> <check out 2.5>
>> ./configure --prefix=/opt/protobuf-2.5 && make && make install
>>
>> <check out 3.0>
>> ./configure --prefix=/opt/protobuf-3.0 && make && make install
>>
>>
>> > On Oct 2, 2016, at 10:41 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>> >
>> > I just installed protobuf 3 on my Mac.
>> >
>> > In case anyone doesn't have protobuf-2.5.0 source locally, please
>> remember
>> > to backup protoc executable before running 'sudo make install' under the
>> > protobuf 3 dir.
>> >
>> > Cheers
>> >
>> > On Sun, Oct 2, 2016 at 10:18 AM, Andrew Purtell <
>> andrew.purtell@gmail.com>
>> > wrote:
>> >
>> >> I have 2.5 and 3.0 installed as /opt/protobuf-<version>, and have
bash
>> >> scripts that add the appropriate version's bin directory to the path.
>> Not
>> >> particularly onerous as I also have to switch between required JDK
>> >> versions, so the scripts also set JAVA_HOME at and add JDK bin to the
>> path
>> >> for the required JDK for the build.
>> >>
>> >> Unlike with the scala compiler, which is after all JVM bytecode at a
>> >> fundamental level, I don't think maven automation for automatic download
>> >> and execution is possible. protoc is a native binary.
>> >>
>> >>> On Oct 1, 2016, at 11:30 PM, 张铎 <palomino219@gmail.com> wrote:
>> >>>
>> >>> Do we need to install protoc 3.0 manully before building HBase or the
>> >> maven
>> >>> protobuf plugin will automatically download the protoc compiler? Maybe
>> we
>> >>> need to install protoc 3.0 in the precommit docker file.
>> >>>
>> >>> 2016-10-02 14:20 GMT+08:00 张铎 <palomino219@gmail.com>:
>> >>>
>> >>>>
>> >>>> 2016-10-02 0:50 GMT+08:00 Stack <stack@duboce.net>:
>> >>>>
>> >>>>>> On Sat, Oct 1, 2016 at 7:20 AM, 张铎 <palomino219@gmail.com>
wrote:
>> >>>>>>
>> >>>>>> Can we setup a compatibility checker job in jenkins? Start
a
>> >>>>> minicluster in
>> >>>>>> one process, and use a client in another process to communicate
with
>> >> it.
>> >>>>>> The version of the client should be >= 0.98 and <=
the version of
>> the
>> >>>>>> minicluster. Of course we need to design the testing code
carefully
>> to
>> >>>>> make
>> >>>>>> sure that we have tested all the cases.
>> >>>>>>
>> >>>>>>
>> >>>>> +1. We need this up and running before we put out an hbase-2.0.0.
I
>> >> know
>> >>>>> Matteo does this test manually on a regular basis but a formalization
>> >>>>> would
>> >>>>> help. I can add an exercise of Coprocessor Endpoints. I believe
this
>> >> is on
>> >>>>> Dima's list of TODOs but will let him speak for himself.
>> >>>>>
>> >>>>>
>> >>>>>> And also I think we should make sure that no proto3 only
feature is
>> >>>>>> introduced in our proto files until branch-1 is dead. Maybe
a
>> >> precommit
>> >>>>>> check?
>> >>>>>>
>> >>>>>>
>> >>>>> I think you mean wire-format breaking changes?  Agree. We have
our
>> PB3
>> >> set
>> >>>>> to 2.5 compat mode and yes, we can't move on from this until
we are
>> in
>> >> a
>> >>>>> place where we can say no to 2.5 clients.
>> >>>>>
>> >>>> Yes, for example, pb2.5 does not support map so we should not use
map
>> in
>> >>>> our proto files.
>> >>>>
>> >>>>>
>> >>>>> Making use of PB3isms cannot be avoided. PB3.1 adds a native
>> >> replacement
>> >>>>> for our HBaseZeroCopyByteString/ByteStringer hack. It also adds
>> >> 'unsafe'
>> >>>>> methods that we need to exploit if we are to keep our read/write
>> paths
>> >>>>> offheap.
>> >>>>>
>> >>>>> St.Ack
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>> Thanks.
>> >>>>>>
>> >>>>>> 2016-10-01 11:55 GMT+08:00 Sean Busbey <busbey@cloudera.com>:
>> >>>>>>
>> >>>>>>> have we experimentally confirmed that wire compatibility
is
>> >>>>>>> maintained? I saw one mention of expecting wire compatibility
to be
>> >>>>>>> fine, but nothing with someone using e.g. the clusterdock
work or
>> >>>>>>> something to mix servers / clients or do replication.
>> >>>>>>>
>> >>>>>>>> On Fri, Sep 30, 2016 at 6:30 PM, Stack <stack@duboce.net>
wrote:
>> >>>>>>>> I intend to do a mass commit late this weekend that
moves us on
>> to a
>> >>>>>>> shaded
>> >>>>>>>> protobuf-3.1.0, either Sunday night or Monday morning.
>> >>>>>>>>
>> >>>>>>>> If objection, please speak up or if need more time
for
>> >>>>>>>> consideration/review, just shout.
>> >>>>>>>>
>> >>>>>>>> I want to merge the branch HBASE-16264 into master
(it is running
>> >>>>> here
>> >>>>>> up
>> >>>>>>>> on jenkins https://builds.apache.org/
>> view/H-L/view/HBase/job/HBASE-
>> >>>>>>> 16264/).
>> >>>>>>>> The branch at HBASE-16264 has three significant
bodies-of-work
>> that
>> >>>>>>>> unfortunately are tangled and can only go in of
a piece.
>> >>>>>>>>
>> >>>>>>>> * HBASE-16264 <https://issues.apache.org/jira/browse/HBASE-16264>
>> >>>>> The
>> >>>>>>>> shading of our protobuf usage so we can upgrade
and/or run with a
>> >>>>>> patched
>> >>>>>>>> protobuf WITHOUT breaking REST, Spark, and in particular,
>> >>>>> Coprocessor
>> >>>>>>>> Endpoints.
>> >>>>>>>> * HBASE-16567 <https://issues.apache.org/jira/browse/HBASE-16567>
>> >>>>> A
>> >>>>>>> move
>> >>>>>>>> up on to (shaded) protobuf-3.1.0
>> >>>>>>>> * HBASE-16741 <https://issues.apache.org/jira/browse/HBASE-16741>
>> >>>>> An
>> >>>>>>>> amendment of our generate protobufs step to include
shading and a
>> >>>>>>> bundling
>> >>>>>>>> of protobuf src (with a means of calling a patch
srcs hook)
>> >>>>>>>>
>> >>>>>>>> Together we're talking about 40MB of change mostly
made of the
>> >>>>> movement
>> >>>>>>> of
>> >>>>>>>> generated files or the application of a pattern
that alters where
>> we
>> >>>>>> get
>> >>>>>>>> imports from. When done, you should notice no difference
and
>> should
>> >>>>> be
>> >>>>>>> able
>> >>>>>>>> to go about your business as per usual. Upside is
that we will be
>> >>>>> able
>> >>>>>> to
>> >>>>>>>> avoid coming onheap doing protobuf marshalling/unmarshalling
as
>> >>>>>> protobuf
>> >>>>>>>> 2.5.0 requires. Downside is that we repeat a good
portion of our
>> >>>>>> internal
>> >>>>>>>> protos, once non-shaded so Coprocessor Endpoints
can keep working
>> >>>>> and
>> >>>>>>> then
>> >>>>>>>> again as shaded for internal use.
>> >>>>>>>>
>> >>>>>>>> I provide some more overview below on the changes.
See the shading
>> >>>>> doc
>> >>>>>>>> here:
>> >>>>>>>> https://docs.google.com/document/d/1H4NgLXQ9Y9KejwobddCqaVMEDCGby
>> >>>>>>> DcXtdF5iAfDIEk/edit#
>> >>>>>>>> for more detail (Patches are up on review board
-- except the
>> latest
>> >>>>>>>> HBASE-16264 which is too big for JIRA and RB). I
am currently
>> >>>>> working
>> >>>>>> on
>> >>>>>>> a
>> >>>>>>>> devs chapter for the book on protobuf going forward
that will go
>> in
>> >>>>> as
>> >>>>>>> part
>> >>>>>>>> of this patch.
>> >>>>>>>>
>> >>>>>>>> Thanks,
>> >>>>>>>> St.Ack
>> >>>>>>>>
>> >>>>>>>> Items of note:
>> >>>>>>>>
>> >>>>>>>> * Two new modules; one named hbase-protocol-shaded
that is used by
>> >>>>>> hbase
>> >>>>>>>> core. It has in it a shaded (and later patched)
protobuf. The
>> other
>> >>>>> new
>> >>>>>>>> module is hbase-endpoint which goes after hbase-server
and has
>> those
>> >>>>>>>> bundled endpoints that I was able to break out of
core (there are
>> a
>> >>>>> few
>> >>>>>>>> that are hopelessly entangled that need to be undone
as CPEPs but
>> >>>>>>>> fortunately belong in core: Auth, Access, MultiRow).
>> >>>>>>>> * I've tested running a branch-1 CPEP against a
master with these
>> >>>>>>> patches
>> >>>>>>>> in place and stuff like ACL (A CPEP) run from the
branch-1 shell
>> >>>>> work
>> >>>>>>>> against the branch-2 server.
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>> On Mon, Aug 22, 2016 at 5:20 PM, Stack <stack@duboce.net>
wrote:
>> >>>>>>>>>
>> >>>>>>>>> This project goes on. I updated HBASE-1563 "Shade
protobuf" with
>> >>>>> some
>> >>>>>>> doc
>> >>>>>>>>> on a final approach. We need to be able to refer
to both shaded
>> and
>> >>>>>>>>> non-shaded protobuf so we can support sending
HDFS old-school pb
>> >>>>>>> Messages
>> >>>>>>>>> but also so Coprocessor Endpoints keep working
though internally
>> >>>>>>> protobufs
>> >>>>>>>>> have been relocated. Funny you should ask, but
yes, there are
>> some
>> >>>>>>>>> downsides (as predicted by contributors on the
JIRA). I'd be
>> >>>>>> interested
>> >>>>>>> to
>> >>>>>>>>> hear if they are too burdensome. In particular,
your IDE
>> experience
>> >>>>>>> gets a
>> >>>>>>>>> little convoluted as you will need to add to
your build path, a
>> jar
>> >>>>>> with
>> >>>>>>>>> the relocated pbs. A pain.
>> >>>>>>>>>
>> >>>>>>>>> Thanks,
>> >>>>>>>>> St.Ack
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>> On Wed, Apr 13, 2016 at 6:09 AM, Stack <stack@duboce.net>
>> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>> On Tue, Apr 12, 2016 at 9:26 PM, Sean Busbey
<busbey@apache.org
>> >
>> >>>>>>> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>>>> On Tue, Apr 12, 2016 at 6:17 PM,
Stack <stack@duboce.net>
>> >> wrote:
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> On an initial pass, the only difficult
part seems to be
>> >>>>>> interaction
>> >>>>>>>>>>> with
>> >>>>>>>>>>>> HDFS in asyncwal (might just pull
in the HDFS messages).
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> I have some idea how we can make this
work either by pushing
>> >>>>>> asyncwal
>> >>>>>>>>>>> upstream to HDFS or through some maven
tricks, depending on how
>> >>>>> much
>> >>>>>>>>>>> time we have.
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> Maven tricks? Tell us more. Here or drop
a note up in the issue.
>> >>>>>>>>>> Thanks Sean,
>> >>>>>>>>>> St.Ack
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> --
>> >>>>>>> busbey
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>>
>> >>
>>

Mime
View raw message