hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anoop John <anoop.hb...@gmail.com>
Subject Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)
Date Sun, 02 Oct 2016 06:11:25 GMT
+1 for the merge.  I had reviewed the big patch which adds new modules and
replace refs to shaded pb path.  Will just see the others also. Thanks for
this great work Stack.

Anoop

On Sunday, October 2, 2016, Stack <stack@duboce.net> wrote:
> On Sat, Oct 1, 2016 at 11:44 AM, Andrew Purtell <andrew.purtell@gmail.com>
> wrote:
>
> Slower? Hmm. For what it's worth after the merge (Monday?) I will take the
>> result through the YCSB workload set with security active and pull out
JFR
>> trace files and GC logs. A Phoenix perf test would also be interesting,
but
>> I'm not sure how much time I'd have getting it to compile against and
>> behave well running on branch-2, so maybe not.
>>
>>
> I just ran the same small-context loading test w/o the patch and perf was
> the same (Poor!  We need to do a bit of work on master branch it looks
> like).
>
> Thanks for the offer Andrew. I'd be interested in whatever you might find.
> I'll be having a go at perf again myself in coming weeks. We can compare
> notes.
>
> St.Ack
>
>
>
>
>> > On Oct 1, 2016, at 10:57 AM, Stack <stack@duboce.net> wrote:
>> >
>> >> On Sat, Oct 1, 2016 at 9:41 AM, Stack <stack@duboce.net> wrote:
>> >>
>> >>
>> >>
>> >>> On Fri, Sep 30, 2016 at 8:55 PM, Sean Busbey <busbey@cloudera.com>
>> wrote:
>> >>>
>> >>> have we experimentally confirmed that wire compatibility is
>> >>> maintained? I saw one mention of expecting wire compatibility to be
>> >>> fine, but nothing with someone using e.g. the clusterdock work or
>> >>> something to mix servers / clients or do replication.
>> >>>
>> >>>
>> >>
>> >> I tried it out a few times in the small: using a 1.2.0 shell to do ACL
>> ops
>> >> against a patched master server; i.e. a branch-1 client Coprocessor
>> >> Endpoint works against a patched branch-2 server. More tests to follow
>> of
>> >> course...
>> >>
>> >>
>> >
>> > I reread the above. You were asking about more than Coprocessor
Endpoint
>> > compatibility (Pardon me; I have been a little fixated on CPEPs of
late).
>> > Yeah, seems fine. I can do load tests from a branch-1 client against my
>> > patched master node. Ram and Anoop have also spent a bunch of time w/
PB3
>> > serialization and claim it compatible. This is apart from the claims by
>> > protobuf crew that it is supposed to be.
>> > Thanks,
>> > M
>> >
>> > P.S. Just retried it just in case and seems to all work (if slow... I
>> need
>> > to look into that)
>> >
>> >
>> >> St.Ack
>> >>
>> >
>> >
>> >>
>> >>
>> >>>> On Fri, Sep 30, 2016 at 6:30 PM, Stack <stack@duboce.net>
wrote:
>> >>>> I intend to do a mass commit late this weekend that moves us on
to a
>> >>> shaded
>> >>>> protobuf-3.1.0, either Sunday night or Monday morning.
>> >>>>
>> >>>> If objection, please speak up or if need more time for
>> >>>> consideration/review, just shout.
>> >>>>
>> >>>> I want to merge the branch HBASE-16264 into master (it is running
here
>> >>> up
>> >>>> on jenkins https://builds.apache.org/view/H-L/view/HBase/job/HBASE-
>> 1626
>> >>> 4/).
>> >>>> The branch at HBASE-16264 has three significant bodies-of-work that
>> >>>> unfortunately are tangled and can only go in of a piece.
>> >>>>
>> >>>> * HBASE-16264 <https://issues.apache.org/jira/browse/HBASE-16264>
The
>> >>>> shading of our protobuf usage so we can upgrade and/or run with
a
>> >>> patched
>> >>>> protobuf WITHOUT breaking REST, Spark, and in particular,
Coprocessor
>> >>>> Endpoints.
>> >>>> * HBASE-16567 <https://issues.apache.org/jira/browse/HBASE-16567>
A
>> >>> move
>> >>>> up on to (shaded) protobuf-3.1.0
>> >>>> * HBASE-16741 <https://issues.apache.org/jira/browse/HBASE-16741>
An
>> >>>> amendment of our generate protobufs step to include shading and
a
>> >>> bundling
>> >>>> of protobuf src (with a means of calling a patch srcs hook)
>> >>>>
>> >>>> Together we're talking about 40MB of change mostly made of the
>> movement
>> >>> of
>> >>>> generated files or the application of a pattern that alters where
we
>> get
>> >>>> imports from. When done, you should notice no difference and should
be
>> >>> able
>> >>>> to go about your business as per usual. Upside is that we will be
able
>> >>> to
>> >>>> avoid coming onheap doing protobuf marshalling/unmarshalling as
>> protobuf
>> >>>> 2.5.0 requires. Downside is that we repeat a good portion of our
>> >>> internal
>> >>>> protos, once non-shaded so Coprocessor Endpoints can keep working
and
>> >>> then
>> >>>> again as shaded for internal use.
>> >>>>
>> >>>> I provide some more overview below on the changes. See the shading
doc
>> >>>> here:
>> >>>> https://docs.google.com/document/d/1H4NgLXQ9Y9KejwobddCqaVME
>> >>> DCGbyDcXtdF5iAfDIEk/edit#
>> >>>> for more detail (Patches are up on review board -- except the latest
>> >>>> HBASE-16264 which is too big for JIRA and RB). I am currently
working
>> >>> on a
>> >>>> devs chapter for the book on protobuf going forward that will go
in
as
>> >>> part
>> >>>> of this patch.
>> >>>>
>> >>>> Thanks,
>> >>>> St.Ack
>> >>>>
>> >>>> Items of note:
>> >>>>
>> >>>> * Two new modules; one named hbase-protocol-shaded that is used
by
>> >>> hbase
>> >>>> core. It has in it a shaded (and later patched) protobuf. The other
>> new
>> >>>> module is hbase-endpoint which goes after hbase-server and has those
>> >>>> bundled endpoints that I was able to break out of core (there are
a
>> few
>> >>>> that are hopelessly entangled that need to be undone as CPEPs but
>> >>>> fortunately belong in core: Auth, Access, MultiRow).
>> >>>> * I've tested running a branch-1 CPEP against a master with these
>> >>> patches
>> >>>> in place and stuff like ACL (A CPEP) run from the branch-1 shell
work
>> >>>> against the branch-2 server.
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>> On Mon, Aug 22, 2016 at 5:20 PM, Stack <stack@duboce.net>
wrote:
>> >>>>>
>> >>>>> This project goes on. I updated HBASE-1563 "Shade protobuf"
with
some
>> >>> doc
>> >>>>> on a final approach. We need to be able to refer to both shaded
and
>> >>>>> non-shaded protobuf so we can support sending HDFS old-school
pb
>> >>> Messages
>> >>>>> but also so Coprocessor Endpoints keep working though internally
>> >>> protobufs
>> >>>>> have been relocated. Funny you should ask, but yes, there are
some
>> >>>>> downsides (as predicted by contributors on the JIRA). I'd be
>> >>> interested to
>> >>>>> hear if they are too burdensome. In particular, your IDE experience
>> >>> gets a
>> >>>>> little convoluted as you will need to add to your build path,
a jar
>> >>> with
>> >>>>> the relocated pbs. A pain.
>> >>>>>
>> >>>>> Thanks,
>> >>>>> St.Ack
>> >>>>>
>> >>>>>
>> >>>>>> On Wed, Apr 13, 2016 at 6:09 AM, Stack <stack@duboce.net>
wrote:
>> >>>>>>
>> >>>>>> On Tue, Apr 12, 2016 at 9:26 PM, Sean Busbey <busbey@apache.org>
>> >>> wrote:
>> >>>>>>
>> >>>>>>>> On Tue, Apr 12, 2016 at 6:17 PM, Stack <stack@duboce.net>
wrote:
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> On an initial pass, the only difficult part seems
to be
>> interaction
>> >>>>>>> with
>> >>>>>>>> HDFS in asyncwal (might just pull in the HDFS messages).
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>> I have some idea how we can make this work either by
pushing
>> asyncwal
>> >>>>>>> upstream to HDFS or through some maven tricks, depending
on how
>> much
>> >>>>>>> time we have.
>> >>>>>>>
>> >>>>>>
>> >>>>>> Maven tricks? Tell us more. Here or drop a note up in the
issue.
>> >>>>>> Thanks Sean,
>> >>>>>> St.Ack
>> >>>>>>
>> >>>>>
>> >>>>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> busbey
>> >>>
>> >>
>> >>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message