hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 张铎 <palomino...@gmail.com>
Subject Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)
Date Sun, 02 Oct 2016 06:20:43 GMT
2016-10-02 0:50 GMT+08:00 Stack <stack@duboce.net>:

> On Sat, Oct 1, 2016 at 7:20 AM, 张铎 <palomino219@gmail.com> wrote:
>
> > Can we setup a compatibility checker job in jenkins? Start a minicluster
> in
> > one process, and use a client in another process to communicate with it.
> > The version of the client should be >= 0.98 and <= the version of the
> > minicluster. Of course we need to design the testing code carefully to
> make
> > sure that we have tested all the cases.
> >
> >
> +1. We need this up and running before we put out an hbase-2.0.0. I know
> Matteo does this test manually on a regular basis but a formalization would
> help. I can add an exercise of Coprocessor Endpoints. I believe this is on
> Dima's list of TODOs but will let him speak for himself.
>
>
> > And also I think we should make sure that no proto3 only feature is
> > introduced in our proto files until branch-1 is dead. Maybe a precommit
> > check?
> >
> >
> I think you mean wire-format breaking changes?  Agree. We have our PB3 set
> to 2.5 compat mode and yes, we can't move on from this until we are in a
> place where we can say no to 2.5 clients.
>
Yes, for example, pb2.5 does not support map so we should not use map in
our proto files.

>
> Making use of PB3isms cannot be avoided. PB3.1 adds a native replacement
> for our HBaseZeroCopyByteString/ByteStringer hack. It also adds 'unsafe'
> methods that we need to exploit if we are to keep our read/write paths
> offheap.
>
> St.Ack
>
>
>
>
>
> > Thanks.
> >
> > 2016-10-01 11:55 GMT+08:00 Sean Busbey <busbey@cloudera.com>:
> >
> > > have we experimentally confirmed that wire compatibility is
> > > maintained? I saw one mention of expecting wire compatibility to be
> > > fine, but nothing with someone using e.g. the clusterdock work or
> > > something to mix servers / clients or do replication.
> > >
> > > On Fri, Sep 30, 2016 at 6:30 PM, Stack <stack@duboce.net> wrote:
> > > > I intend to do a mass commit late this weekend that moves us on to a
> > > shaded
> > > > protobuf-3.1.0, either Sunday night or Monday morning.
> > > >
> > > > If objection, please speak up or if need more time for
> > > > consideration/review, just shout.
> > > >
> > > > I want to merge the branch HBASE-16264 into master (it is running
> here
> > up
> > > > on jenkins https://builds.apache.org/view/H-L/view/HBase/job/HBASE-
> > > 16264/).
> > > > The branch at HBASE-16264 has three significant bodies-of-work that
> > > > unfortunately are tangled and can only go in of a piece.
> > > >
> > > >  * HBASE-16264 <https://issues.apache.org/jira/browse/HBASE-16264>
> The
> > > > shading of our protobuf usage so we can upgrade and/or run with a
> > patched
> > > > protobuf WITHOUT breaking REST, Spark, and in particular, Coprocessor
> > > > Endpoints.
> > > >  * HBASE-16567 <https://issues.apache.org/jira/browse/HBASE-16567>
A
> > > move
> > > > up on to (shaded) protobuf-3.1.0
> > > >  * HBASE-16741 <https://issues.apache.org/jira/browse/HBASE-16741>
> An
> > > > amendment of our generate protobufs step to include shading and a
> > > bundling
> > > > of protobuf src (with a means of calling a patch srcs hook)
> > > >
> > > > Together we're talking about 40MB of change mostly made of the
> movement
> > > of
> > > > generated files or the application of a pattern that alters where we
> > get
> > > > imports from. When done, you should notice no difference and should
> be
> > > able
> > > > to go about your business as per usual. Upside is that we will be
> able
> > to
> > > > avoid coming onheap doing protobuf marshalling/unmarshalling as
> > protobuf
> > > > 2.5.0 requires. Downside is that we repeat a good portion of our
> > internal
> > > > protos, once non-shaded so Coprocessor Endpoints can keep working and
> > > then
> > > > again as shaded for internal use.
> > > >
> > > > I provide some more overview below on the changes. See the shading
> doc
> > > > here:
> > > > https://docs.google.com/document/d/1H4NgLXQ9Y9KejwobddCqaVMEDCGby
> > > DcXtdF5iAfDIEk/edit#
> > > > for more detail (Patches are up on review board -- except the latest
> > > > HBASE-16264 which is too big for JIRA and RB). I am currently working
> > on
> > > a
> > > > devs chapter for the book on protobuf going forward that will go in
> as
> > > part
> > > > of this patch.
> > > >
> > > > Thanks,
> > > > St.Ack
> > > >
> > > > Items of note:
> > > >
> > > >  * Two new modules; one named hbase-protocol-shaded that is used by
> > hbase
> > > > core. It has in it a shaded (and later patched) protobuf. The other
> new
> > > > module is hbase-endpoint which goes after hbase-server and has those
> > > > bundled endpoints that I was able to break out of core (there are a
> few
> > > > that are hopelessly entangled that need to be undone as CPEPs but
> > > > fortunately belong in core: Auth, Access, MultiRow).
> > > >  * I've tested running a branch-1 CPEP against a master with these
> > > patches
> > > > in place and stuff like ACL (A CPEP) run from the branch-1 shell work
> > > > against the branch-2 server.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, Aug 22, 2016 at 5:20 PM, Stack <stack@duboce.net> wrote:
> > > >
> > > >> This project goes on. I updated HBASE-1563 "Shade protobuf" with
> some
> > > doc
> > > >> on a final approach. We need to be able to refer to both shaded and
> > > >> non-shaded protobuf so we can support sending HDFS old-school pb
> > > Messages
> > > >> but also so Coprocessor Endpoints keep working though internally
> > > protobufs
> > > >> have been relocated. Funny you should ask, but yes, there are some
> > > >> downsides (as predicted by contributors on the JIRA). I'd be
> > interested
> > > to
> > > >> hear if they are too burdensome. In particular, your IDE experience
> > > gets a
> > > >> little convoluted as you will need to add to your build path, a jar
> > with
> > > >> the relocated pbs. A pain.
> > > >>
> > > >> Thanks,
> > > >> St.Ack
> > > >>
> > > >>
> > > >> On Wed, Apr 13, 2016 at 6:09 AM, Stack <stack@duboce.net> wrote:
> > > >>
> > > >>> On Tue, Apr 12, 2016 at 9:26 PM, Sean Busbey <busbey@apache.org>
> > > wrote:
> > > >>>
> > > >>>> On Tue, Apr 12, 2016 at 6:17 PM, Stack <stack@duboce.net>
wrote:
> > > >>>> >
> > > >>>> >
> > > >>>> > On an initial pass, the only difficult part seems to
be
> > interaction
> > > >>>> with
> > > >>>> > HDFS in asyncwal (might just pull in the HDFS messages).
> > > >>>> >
> > > >>>> >
> > > >>>>
> > > >>>> I have some idea how we can make this work either by pushing
> > asyncwal
> > > >>>> upstream to HDFS or through some maven tricks, depending on
how
> much
> > > >>>> time we have.
> > > >>>>
> > > >>>
> > > >>> Maven tricks? Tell us more. Here or drop a note up in the issue.
> > > >>> Thanks Sean,
> > > >>> St.Ack
> > > >>>
> > > >>
> > > >>
> > >
> > >
> > >
> > > --
> > > busbey
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message