ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Emmanouil Gkatziouras <gkatzio...@gmail.com>
Subject Re: [DISCUSS] Pub/Sub Streamer Implementation
Date Sun, 13 Oct 2019 23:10:47 GMT
Hi @Saikat Maitra

I have my implementation on an Ignite fork on my personal GitHub account [1]
It is based on the Kafka implementation which is of a polling nature.
GCP does not provide a service like Kafka. Instead it provides us with
Pub/Sub which is just like Kafka.
Therefore using the Kafka streamer as a guideline made more sense to me.

Apart from the implementation I have also added a unit test.
It definitely needs some polishing and a double check on my side to make
the review smooth.
For the tests a local Pub/Sub [2] server has to be setup however I am not
sure which is the best way to do so with regards to your CI/CD.

[1] https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub
[2] https://egkatzioura.com/2019/09/22/pub-sub-local-emulator/

*Emmanouil Gkatziouras*
https://egkatzioura.com/ | https://www.linkedin.com/in/gkatziourasemmanouil/
https://github.com/gkatzioura


On Sat, 12 Oct 2019 at 02:11, Saikat Maitra <saikat.maitra@gmail.com> wrote:

> Hello Emmanouil,
>
> Can you please share more info about the implementation?
>
> We have similar implementation for data streamers with Kafka and RocketMQ
>
> https://apacheignite-mix.readme.io/docs/kafka-streamer
>
> https://apacheignite-mix.readme.io/docs/rocketmq-streamer
>
> Please review and share if you have any questions.
>
> Regards,
> Saikat
>
>
>
> On Fri, Oct 11, 2019 at 4:56 PM Emmanouil Gkatziouras <
> gkatzioura@gmail.com>
> wrote:
>
> > Hi all,
> >
> > By mentioning Pub/Sub I mean Google Cloud's Pub/Sub [1]
> > Yes it is indeed cloud specific. It is the messaging service provided by
> > GCP.
> > I have made the integration with it, but I haven't published it yet (I am
> > waiting for the discussion/ticket and proceed with the contribution).
> >
> > Kind regards,
> > Emmanouil
> >
> > [1] https://cloud.google.com/pubsub/docs/overview
> >
> > *Emmanouil Gkatziouras*
> > https://egkatzioura.com/ |
> > https://www.linkedin.com/in/gkatziourasemmanouil/
> > https://github.com/gkatzioura
> >
> >
> > On Fri, 11 Oct 2019 at 21:45, Denis Magda <dmagda@apache.org> wrote:
> >
> > > Hello Emmanouil,
> > >
> > > Are you referring to any specific implementation tailored for a cloud?
> If
> > > I'm not mistaken you were mentioning AWS/GCE. Is there any Github
> source
> > > code?
> > >
> > > Here is a list of existing integrations with streaming and pub/sub
> > > frameworks:
> > > https://apacheignite-mix.readme.io/docs/overview
> > >
> > > with the source code in Ignite master:
> > > https://github.com/apache/ignite/tree/master/modules
> > >
> > > Check out those resources to see what are the most suitable Ignite APIs
> > for
> > > such integrations.
> > >
> > > @Saikat Maitra <saikat.maitra@gmail.com> , @Roman <rshtykh@yahoo.com>
> ,
> > as
> > > streaming technologies contributors/maintainers, could you please join
> > this
> > > thread?
> > >
> > > -
> > > Denis
> > >
> > >
> > > On Thu, Oct 10, 2019 at 7:58 AM Emmanouil Gkatziouras <
> > > gkatzioura@gmail.com>
> > > wrote:
> > >
> > > > Hello everyone,
> > > >
> > > > I started using Ignite lately. Part of my work involves Pub/Sub,
> thus I
> > > > created a pub/sub streamer.
> > > > Pub/Sub is pretty close to Kafka.
> > > > Here's how the streamer works. Every node creates an instance of a
> > > Pub/Sub
> > > > client.
> > > > The clients subscribe to the same topic using the same subscription
> > name.
> > > > A message is being sent and one of the nodes will receive the
> message,
> > > > process it and then forward it to Ignite using the extractor
> provided.
> > > >
> > > > Any thoughts?
> > > >
> > > > *Emmanouil Gkatziouras*
> > > > https://egkatzioura.com/ |
> > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > > > https://github.com/gkatzioura
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message