activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Martyn Taylor <mtay...@redhat.com>
Subject Re: Kafka ServiceConnector
Date Fri, 27 Oct 2017 10:05:17 GMT
Looking at the Kafka Connect example.  I see that the integration
components are all maintained outside of the Kafka project.  They're listed
here: https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem

How about we do something similar and create a wiki/web page linking the
integration projects.  I realise our website isn't the most accessible atm
but going forward this should be improved.

On Thu, Oct 26, 2017 at 5:42 PM, Robbie Gemmell <robbie.gemmell@gmail.com>
wrote:

> The related PR is https://github.com/apache/activemq-artemis/pull/1607
> for anyone not following along thus far, or later reference.
>
> My main view there was that such an extension would be better
> maintained seperate from the main broker repository+distribution for a
> variety of reasons. After that I didn't have a strong view on where it
> lives, but I would say that I don't actually see any issue at all with
> it being outwith the ActiveMQ project. To me thats just a key thing
> enabled by such plugin points existing, and not uncommon with other
> projects/components.
>
> Robbie
>
> On 26 October 2017 at 16:46, Michael André Pearce
> <michael.andre.pearce@me.com> wrote:
> > As noted on the PR part of the PR discussion is should Service
> Connectors exist with the broker code.
> >
> > Service Connectors / Extensions are ActiveMQ Artemis specific.
> >
> > I think having them within the project space is good so we can grow an
> eco system of some,  people can just use will aid in having better broker
> adoption as tools people look for would be available if contributed.
> >
> > As stated above it seems the general consensus on the PR is though liked
> is the broker code base best for them to live?
> >
> > If we don’t wish them to exist with broker code so they can have their
> own release cycle, and don’t murky the water of the broker, I would like to
> discuss if we could make a sub project for these to live, and possibly
> propose a sub project be setup called “ActiveMQ Artemis Integrations” ?
> >
> > If setup I would also propose maybe moving the spring integration there
> also.
> >
> > Cheers
> > Mike
> >
> >
> >
> >
> >
> >
> >
> > Sent from my iPhone
> >
> >> On 6 Oct 2017, at 12:20, Michael André Pearce <
> michael.andre.pearce@me.com> wrote:
> >>
> >> Yes and no, it would be same as arguing that core bridge between
> artemis clusters could also be separated components.
> >>
> >> The idea is to effectively have similar to a core bridge but to kafka.
> >>
> >> Some key parts why are (and probably similar to why core bridge is
> implemented within the broker)
> >>
> >> Latency
> >> Extra processes and hops
> >> Out the box in broker support
> >>
> >> Cheers
> >> Mike
> >>
> >>
> >> Sent from my iPhone
> >>
> >>> On 6 Oct 2017, at 12:14, Andy Taylor <andy.tayls67@gmail.com> wrote:
> >>>
> >>> I think https://github.com/ppatierno/kafka-connect-amqp is an example
> of
> >>> this
> >>>
> >>> On 6 October 2017 at 12:07, Christopher Shannon <
> >>> christopher.l.shannon@gmail.com> wrote:
> >>>
> >>>> Why not use Kafka's Connect framework?  Seems like building a source
> >>>> connector here would make the most sense.  This is not to say I'm
> against
> >>>> the idea I'm just wondering the benefits of doing it from the Artemis
> side
> >>>> of things when Kafka already has a framework that can be used to
> >>>> import/export data.
> >>>>
> >>>> On Fri, Oct 6, 2017 at 6:16 AM, Michael André Pearce <
> >>>> michael.andre.pearce@me.com> wrote:
> >>>>
> >>>>> Hi Andy,
> >>>>>
> >>>>> As you may be aware or not we run both activemq and kafka each has
a
> >>>>> specific plus point to it, use the right tool for the right job.
> >>>>>
> >>>>> We use activemq as our feature rich, fully jms compliant and
> performant
> >>>>> broker.
> >>>>>
> >>>>> And use kafka more for post processing and data streaming, with
> >>>> historical
> >>>>> feed also.
> >>>>>
> >>>>> Idea here is that producers and consumers bind into our activemq
> where
> >>>> jms
> >>>>> api is wanted or needed, lower per message latencies etc. but then
> we can
> >>>>> bridge messages from a core queue into kafka so that the data can
> flow to
> >>>>> it as fast as possible and consumed from there also.
> >>>>>
> >>>>> Idea is to try make a hybrid taking best of both.
> >>>>>
> >>>>> Cheers
> >>>>> Mike
> >>>>>
> >>>>> Sent from my iPhone
> >>>>>
> >>>>>> On 6 Oct 2017, at 09:37, Andy Taylor <andy.tayls67@gmail.com>
> wrote:
> >>>>>>
> >>>>>> Whats the use case here Michael, sounds like a cool idea and
> something
> >>>>> that
> >>>>>> has been mentioned by different people but I am yet to understand
> how
> >>>>> this
> >>>>>> could be used.
> >>>>>>
> >>>>>> On 6 October 2017 at 07:48, Michael André Pearce <
> >>>>>> michael.andre.pearce@me.com> wrote:
> >>>>>>
> >>>>>>> Hi All,
> >>>>>>>
> >>>>>>> I am looking to contribute an artemis ServiceConnector that
would
> >>>>> leverage
> >>>>>>> the existing service connector interface to bridge from
a queue
> into a
> >>>>>>> kafka topic similar to bridge between artemis clusters.
> >>>>>>>
> >>>>>>> Idea is to make it configurable how to transform and encode
the
> >>>> message
> >>>>> in
> >>>>>>> kafka but as amqp is becoming very well supported by artemis
> provide
> >>>> out
> >>>>>>> the box that the message be encoded in amqp byte protocol.
> >>>>>>>
> >>>>>>> It something I’ve been working on privately for a bit,
but since
> >>>>> becoming
> >>>>>>> a committer here is really like to share it.
> >>>>>>>
> >>>>>>> I hope if I get a positive response here to get a branch
ready
> soon,
> >>>> but
> >>>>>>> feedback if this is welcomed would be great.
> >>>>>>>
> >>>>>>> Cheers
> >>>>>>> Mike
> >>>>>>>
> >>>>>>>
> >>>>>>> Sent from my iPhone
> >>>>>
> >>>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message