hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: [DISCUSS] Kafka Connection, HBASE-15320
Date Fri, 02 Nov 2018 15:25:49 GMT
On Fri, Nov 2, 2018 at 8:20 AM Stack <stack@duboce.net> wrote:

> On Fri, Nov 2, 2018 at 7:30 AM Josh Elser <elserj@apache.org> wrote:
>
>>
>> ...
> Back and forth was dumped into HBASE-21002 (We changed this config on
> hbase-operator-tools config to not do this? I should look).
>
>
I took a look. We asked infra to send gitbox mail to issues@ (they used to
go to dev@). We left the git conversation as JIRA comments option in place
for now (until we get annoyed) [1].

S
1. https://issues.apache.org/jira/browse/INFRA-16886



> Regards pull requests, etc., email configs., etc., we are in experimental
> mode around all this stuff trying to figure it out so
> suggestions/help/exercising the possibilities are all welcome.
>
> Thanks,
> S
>
>
>
>> - Josh
>>
>> On 10/31/18 6:43 PM, Stack wrote:
>> > To tie-off this thread, this nice feature was just pushed on
>> > hbase-connector. See
>> > https://github.com/apache/hbase-connectors/tree/master/kafka for
>> how-to.
>> > Review and commentary welcome.
>> >
>> > Thanks,
>> > S
>> >
>> > On Fri, Aug 3, 2018 at 6:32 AM Hbase Janitor <hbasejanitor@gmail.com>
>> wrote:
>> >
>> >> I opened hbase-21002 to start the scripts and assembly.
>> >>
>> >> Mike
>> >>
>> >> On Thu, Aug 2, 2018, 19:29 Stack <stack@duboce.net> wrote:
>> >>
>> >>> Up in https://issues.apache.org/jira/browse/HBASE-20934 I created an
>> >>> hbase-connectors repo. I put some form on it using the v19 patch from
>> >>> HBASE-15320 "HBase connector for Kafka Connect". It builds and tests
>> >>> pass. Here are some remaining TODOs:
>> >>>
>> >>>   * Figure how to do start scripts: e.g. we need to start up the kafka
>> >>> proxy. It wants some hbase jars, conf dir, and others on the CLASSPATH
>> >>> (Depend on an HBASE_HOME and then source bin/hbase?)
>> >>>   * Can any of the connectors make-do with the shaded client?
>> >>>   * Make connectors standalone or have them share conf, bin, etc?
>> >>>   * Need to do an assembly. Not done.
>> >>>   * Move over REST and thrift next. Mapreduce after?
>> >>>
>> >>> The poms could do w/ a review. Hacked them over from hbase-thirdparty.
>> >>>
>> >>> File issues and apply patches up in JIRA if your up for any of the
>> above.
>> >>>
>> >>> Thanks,
>> >>> S
>> >>>
>> >>> On Wed, Jul 25, 2018 at 10:46 PM Stack <stack@duboce.net> wrote:
>> >>>>
>> >>>>
>> >>>> On Tue, Jul 24, 2018 at 10:01 PM Misty Linville <misty@apache.org>
>> >>> wrote:
>> >>>>>
>> >>>>> I like the idea of a separate connectors repo/release vehicle,
but
>> >> I'm a
>> >>>>> little concerned about the need to release all together to update
>> just
>> >>> one
>> >>>>> of the connectors. How would that work? What kind of compatibility
>> >>>>> guarantees are we signing up for?
>> >>>>>
>> >>>>
>> >>>> I hate responses that begin "Good question" -- so fawning -- but,
>> ahem,
>> >>> good question Misty (in the literal, not flattering, sense).
>> >>>>
>> >>>> I think hbase-connectors will be like hbase-thirdparty. The latter
>> >>> includes netty, pb, guava and a few other bits and pieces so yeah,
>> >>> sometimes a netty upgrade or an improvement on our patch to pb will
>> >> require
>> >>> us releasing all though we are fixing one lib only. Usually, if
>> bothering
>> >>> to make a release, we'll check for fixes or updates we can do in the
>> >> other
>> >>> bundled components.
>> >>>>
>> >>>> On the rate of releases, I foresee a flurry of activity around launch
>> >> as
>> >>> we fill missing bits and address critical bug fixes, but that then it
>> >> will
>> >>> settle down to be boring, with just the occasional update. Thrift and
>> >> REST
>> >>> have been stable for a good while now (not saying this is a good
>> thing).
>> >>> Our Sean just suggested moving mapreduce to connectors too -- an
>> >>> interesting idea -- and this has also been stable too (at least until
>> >>> recently with the shading work). We should talk about the Spark
>> connector
>> >>> when it comes time. It might not be as stable as the others.
>> >>>>
>> >>>> On the compatibility guarantees, we'll semver it so if an
>> incompatible
>> >>> change in a connector or if the connectors have to change to match a
>> new
>> >>> version of hbase, we'll make sure the hbase-connector version number
>> is
>> >>> changed appropriately. On the backend, what Mike says; connectors use
>> >> HBase
>> >>> Public APIs (else they can't be moved to the hbase-connector repo).
>> >>>>
>> >>>> S
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>>
>> >>>>> On Tue, Jul 24, 2018, 9:41 PM Stack <stack@duboce.net>
wrote:
>> >>>>>
>> >>>>>> Grand. I filed https://issues.apache.org/jira/browse/HBASE-20934.
>> >>> Let me
>> >>>>>> have a go at making the easy one work first (the kafka proxy).
Lets
>> >>> see how
>> >>>>>> it goes. I'll report back here.
>> >>>>>> S
>> >>>>>>
>> >>>>>> On Tue, Jul 24, 2018 at 2:43 PM Sean Busbey <busbey@apache.org>
>> >>> wrote:
>> >>>>>>
>> >>>>>>> Key functionality for the project's adoption should
be in the
>> >>> project.
>> >>>>>>> Please do not suggest we donate things to Bahir.
>> >>>>>>>
>> >>>>>>> I apologize if this is brisk. I have had previous negative
>> >>> experiences
>> >>>>>>> with folks that span our communities trying to move
work I spent a
>> >>> lot
>> >>>>>>> of time contributing to within HBase over to Bahir in
an attempt
>> >> to
>> >>>>>>> bypass an agreed upon standard of quality.
>> >>>>>>>
>> >>>>>>> On Tue, Jul 24, 2018 at 3:38 PM, Artem Ervits <
>> >>> artemervits@gmail.com>
>> >>>>>>> wrote:
>> >>>>>>>> Why not just donating the connector to http://bahir.apache.org/
>> >> ?
>> >>>>>>>>
>> >>>>>>>> On Tue, Jul 24, 2018, 12:51 PM Lars Francke <
>> >>> lars.francke@gmail.com>
>> >>>>>>> wrote:
>> >>>>>>>>
>> >>>>>>>>> I'd love to have the Kafka Connector included.
>> >>>>>>>>>
>> >>>>>>>>> @Mike thanks so much for the contribution (and
your planned
>> >> ones)
>> >>>>>>>>>
>> >>>>>>>>> I'm +1 on adding it to the core but I'm also
+1 on having a
>> >>> separate
>> >>>>>>>>> repository under Apache governance
>> >>>>>>>>>
>> >>>>>>>>> On Tue, Jul 24, 2018 at 6:01 PM, Josh Elser
<elserj@apache.org
>> >>>
>> >>>>>> wrote:
>> >>>>>>>>>
>> >>>>>>>>>> +1 to the great point by Duo about use of
non-IA.Public
>> >> classes
>> >>>>>>>>>>
>> >>>>>>>>>> +1 for Apache for the governance (although,
I wouldn't care
>> >> if
>> >>> we
>> >>>>>> use
>> >>>>>>>>>> Github PRs to try to encourage more folks
to contribute), a
>> >>> repo
>> >>>>>> with
>> >>>>>>> the
>> >>>>>>>>>> theme of "connectors" (to include Thrift,
REST, and the
>> >> like).
>> >>> Spark
>> >>>>>>> too
>> >>>>>>>>> --
>> >>>>>>>>>> I think we had suggested that prior, but
it could be a mental
>> >>>>>>> invention
>> >>>>>>>>> of
>> >>>>>>>>>> mine..
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> On 7/24/18 10:16 AM, Hbase Janitor wrote:
>> >>>>>>>>>>
>> >>>>>>>>>>> Hi everyone,
>> >>>>>>>>>>>
>> >>>>>>>>>>> I'm the author of the patch.  A separate
repo for all the
>> >>>>>> connectors
>> >>>>>>> is
>> >>>>>>>>> a
>> >>>>>>>>>>> great idea! I can make whatever changes
necessary to the
>> >>> patch to
>> >>>>>>> help.
>> >>>>>>>>>>>
>> >>>>>>>>>>> I have several other integration type
projects like this
>> >>> planned.
>> >>>>>>>>>>>
>> >>>>>>>>>>> Mike
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> On Tue, Jul 24, 2018, 00:03 Mike Drob
<mdrob@apache.org>
>> >>> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>> I would be ok with all of the connectors
in a single repo.
>> >>> Doing a
>> >>>>>>> repo
>> >>>>>>>>>>>> per
>> >>>>>>>>>>>> connector seems like a large amount
of overhead work.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> On Mon, Jul 23, 2018, 9:12 PM Clay
B. <cwb@clayb.net>
>> >> wrote:
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> [Non-binding]
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> I am all for the Kafka Connect(er)
as indeed it makes
>> >> HBase
>> >>> "more
>> >>>>>>>>>>>>> relevant" and generates buzz
to help me sell HBase
>> >> adoption
>> >>> in my
>> >>>>>>>>>>>>> endeavors.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Also, I would like to see a
connectors repo a lot as I
>> >> would
>> >>>>>>> expect it
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>> can
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>> make the HBase source and releases
more obvious in what is
>> >>>>>>> changing.
>> >>>>>>>>> Not
>> >>>>>>>>>>>>> to distract from Kafka, but
Spark has in the past been a
>> >>> hang-up
>> >>>>>>> and
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>> seems
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>> a good fit in such a repo too;
as such, I would prefer
>> >>> Apache
>> >>>>>> over
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>> GitHub.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> -Clay
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> On Mon, 23 Jul 2018, Andrew
Purtell wrote:
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Would we make a new repo called
hbase-connectors and move
>> >>> REST,
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> thrift,
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>> and this new patch there?
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> I like this idea. We are
already releasing
>> >>> hbase-thirdparty like
>> >>>>>>>>> this.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> On Mon, Jul 23, 2018 at
5:47 PM Stack <stack@duboce.net>
>> >>> wrote:
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> (Thanks for the good discussion)
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Where we think  'outside
of HBase' would be?
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Github seems too 'remote'
from project and from Apache?
>> >>> Would
>> >>>>>> we
>> >>>>>>>>> make
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> a
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>> new
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>> repo called hbase-connectors
and move REST, thrift, and
>> >>> this new
>> >>>>>>>>> patch
>> >>>>>>>>>>>>>>> there?
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Thanks,
>> >>>>>>>>>>>>>>> S
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> On Mon, Jul 23, 2018
at 3:50 PM Josh Elser <
>> >>> elserj@apache.org>
>> >>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> I'm -0 for including
this into the main hbase tree. I
>> >>> feel like
>> >>>>>>>>> we've
>> >>>>>>>>>>>>>>>> made a bit of progress
in cleaning up our core, and
>> >> this
>> >>>>>>> strikes me
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> as
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>> a
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>> step in the wrong direction.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> At the same time,
the integration seems nice enough
>> >> (for
>> >>> the
>> >>>>>>> same
>> >>>>>>>>>>>>>>>> reasons Andrew points
out). Is there a reason this
>> >>> couldn't
>> >>>>>>> exist
>> >>>>>>>>>>>>>>>> outside of HBase
(at the ASF or otherwise)? Given a
>> >> quick
>> >>>>>>> glance at
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> the
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>> patch, it would be quite trivial
to keep separate (just
>> >>> requires
>> >>>>>>> some
>> >>>>>>>>>>>>>>>> heavier scripting
to get it off the ground that the
>> >> HBase
>> >>>>>>> scripts
>> >>>>>>>>> do
>> >>>>>>>>>>>>>>>> setup for). I feel
like that will decrease our debt
>> >>> while we
>> >>>>>>> see if
>> >>>>>>>>>>>>>>>> people start using
it. Our API should be more than
>> >> stable
>> >>>>>>> enough to
>> >>>>>>>>>>>>>>>> prevent any worry
about drift happening from core to
>> >> this
>> >>>>>>> project.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> On 7/23/18 6:35
PM, Stack wrote:
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> We have a very
nice contrib sitting up in HBASE-15320
>> >>> which
>> >>>>>>> via a
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> proxy
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>> --
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> so minimal dependencies
-- adds source and sink for
>> >>> Kafka
>> >>>>>>> Connect.
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> It
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>> is
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> nicely contained
inside two new hbase-kafka-* modules.
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> We good w/ taking
on this new feature?
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> It looks good
to me. Check it out up on HBASE-15320. I
>> >>> was
>> >>>>>>> going
>> >>>>>>>>> to
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> commit
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> to tip of branch-2
so it'd show up in hbase-2.2.x
>> >>> unless you
>> >>>>>>> all
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> want
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>> some
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> backporting
action going on.
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> S
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> --
>> >>>>>>>>>>>>>> Best regards,
>> >>>>>>>>>>>>>> Andrew
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Words like orphans lost
among the crosstalk, meaning torn
>> >>> from
>> >>>>>>>>> truth's
>> >>>>>>>>>>>>>> decrepit hands
>> >>>>>>>>>>>>>>     - A23, Crosstalk
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>
>> >>
>> >
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message