ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saikat Maitra <saikat.mai...@gmail.com>
Subject Re: [DISCUSS] Pub/Sub Streamer Implementation
Date Sat, 19 Oct 2019 02:03:18 GMT
Hello Denis,

Sure, sounds good. I will create a separate discussion thread to get
community feedback on whether we
should join the Bahir or kick-off "Ignite Extensions" project.

Regards,
Saikat

On Thu, Oct 17, 2019 at 2:21 PM Denis Magda <dmagda@apache.org> wrote:

> Folks,
>
> The concept of Apache Bahir is precisely what we need! Saikat, thanks for
> doing the research. Why I believe the * idea* fits us well:
>
>    - All integrations can be stored in separate Github repositories and
>    have their dev lifecycles. We've not obliged to couple all the
> integrations
>    in a single repo. For instance, Spark can be located in a dedicated repo
>    while streaming integrations might be in a single one. It's up to us.
>    - All the repositories and integrations belong to ASF. We're not dumping
>    them on Github but rather keep supporting and developing in accordance
> with
>    ASF vision and practices.
>    - There is a way to reward contributors via committership and PMC
>    membership. You won't get this if a project is just one of the millions
> of
>    Github projects.
>
> Saikat, as Ignite PMC member, could you please kick-off a separate dev-list
> discussion to involve more community members and referring to this thread?
> I think the primary question the community needs to answer whether we
> should join the Bahir or kick-off "Ignite Extensions" project?
>
> -
> Denis
>
>
> On Thu, Oct 17, 2019 at 2:54 AM Alexey Zinoviev <zaleslaw.sin@gmail.com>
> wrote:
>
> > Maybe we could move all our Streaming Integrations there, but what is
> about
> > maintaining and committer permissions to the new repositories?
> > I see the list of the committers and PMC members there
> > https://bahir.apache.org/community-members/
> >
> > Could somebody from Ignite community be added to this list as a
> > PMC/committer to maintain Ignite integrations?
> > If yes, I'd happy to join this project and collaborate with these guys
> > together
> > If no, and we should start from the zero level with external PRs - hmmm,
> > it's better to have our own external repository with ApacheBahirr
> ideology.
> >
> > Also, I agree, that all connectors could be moved there (in ApacheBahirr
> > like repository), but not all modules from the page
> >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> > because
> > they use different parts of core modules and could be developed with
> > different velocity.
> >
> > In reality, before creation of new repositories we need wide discussion
> > based on the document mentioned above.
> >
> > P.S. Of course, we could experiment in the next release 2.8 with one or
> two
> > integrations like twitter or ZeroMQ
> > Also, I'd like Denis Magda idea for the separate repositories in Apache
> > Github to maintain them by anybody.
> >
> >
> > чт, 17 окт. 2019 г. в 05:02, Saikat Maitra <saikat.maitra@gmail.com>:
> >
> > > Hi Denis, Emmanouil
> > >
> > > Thank you for your email. I think creating a separate integration
> project
> > > in github and also proposing it as an apache incubator project will be
> a
> > > good move. The other separate integration modules
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> > > can
> > > be moved to this new apache incubator project.
> > >
> > > There are similar integration available for Flink and Spark in Apache
> > Bahir
> > > https://bahir.apache.org/
> > >
> > > Do you think we should reach out to Apache Bahir community with a
> > proposal
> > > or we should plan to create a separate Apache Incubator project?
> > >
> > > Regards,
> > > Saikat
> > >
> > >
> > >
> > > On Wed, Oct 16, 2019 at 6:21 AM Emmanouil Gkatziouras <
> > > gkatzioura@gmail.com>
> > > wrote:
> > >
> > > > Hi all!
> > > >
> > > > Based on the discussions most probably this is going to be on another
> > > > GitHub repo. Therefore I will prepare a standalone project with the
> > > Pub/Sub
> > > > module and once the repository is created a pull request shall be
> > issued.
> > > > If there is anything else I can do in order to facilitate this please
> > let
> > > > me know.
> > > >
> > > > If a ticket is created for this issue, I shall be able to create a
> pull
> > > > request and thus a build will take place on ignite's CI.
> > > > If everyone is aligned with adding a Pub/Sub implementation may I
> > create
> > > a
> > > > ticket on JIRA?
> > > >
> > > > My next question has to do on how Ignite configures the build jobs in
> > > Team
> > > > City. The unit tests for my implementation need a local Pub/Sub
> server
> > up
> > > > and running (they provide one for testing purposes). Will this take
> > > effect
> > > > in a form of a build script inside my implementation or is it
> something
> > > > that should be configured on Team City?
> > > >
> > > > Kind regards,
> > > > Emmanouil
> > > >
> > > >
> > > > *Emmanouil Gkatziouras*
> > > > https://egkatzioura.com/ |
> > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > > > https://github.com/gkatzioura
> > > >
> > > >
> > > > On Tue, 15 Oct 2019 at 23:27, Denis Magda <dmagda@apache.org> wrote:
> > > >
> > > > > Emmanouil, Saikat,
> > > > >
> > > > > After contemplating on this, let me propose another way with this
> > > > > particular integration that is aligned with our modularization
> > > endeavor.
> > > > > The modularization [1] defines the Ignite core, Ignite modules, and
> > > > > independent integrations. The first two are to be located in Ignite
> > > > > repositories and to be tested/updated by the community for every
> > > release.
> > > > > The modules are a subset of existing integrations that are adopted
> > > widely
> > > > > and require community attention.
> > > > >
> > > > > As for the independent integrations [1], those are supposed to be
> > > located
> > > > > in a separate GitHub repo. I would advise us to start implementing
> > this
> > > > > concept with the Pub/Sub integration. Even though it will be
> located
> > > in a
> > > > > separate Github repo, it will be listed among officially available
> > > > > integrations on the Ignite website featuring Emmanouil as a
> > > contributor.
> > > > >
> > > > > What do you think?
> > > > >
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> > > > >
> > > > >
> > > > >
> > > > > -
> > > > > Denis
> > > > >
> > > > >
> > > > > On Mon, Oct 14, 2019 at 9:47 PM Saikat Maitra <
> > saikat.maitra@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Emmanouil,
> > > > > >
> > > > > > The changes looks good to me. I wanted to check if you are also
> > able
> > > to
> > > > > > validate that the once data is added to Ignite cluster you are
> also
> > > > able
> > > > > to
> > > > > > access it using another Ignite client or using rest endpoints.
> > > > > >
> > > > > > We use Teamcity for CI process, the details are as follows
> > > > > >
> > > > > > Trigger validation of those test suites that have been affected
> by
> > > your
> > > > > > changes on TeamCity <http://ci.ignite.apache.org/>: <
> > > > > http://204.14.53.153/
> > > > > > >
> > > > > >
> > > > > >    - Locate a test suite you have to check, press button named
> > "..."
> > > > that
> > > > > >    is located on the left of "Run" button. "Run custom build"
> > dialog
> > > > will
> > > > > >    appear;
> > > > > >    - Go to "Changes" tab and choose
> > "pull/<pull-request-number>/head"
> > > > in
> > > > > >    "Build branch" dropdown list;
> > > > > >    - Press "Run build" button and monitor tests results.
> > > > > >
> > > > > >
> > > > > >
> > > > > > @Roman rshtykh@yahoo.com - can you also please take a look?
> > > > > >
> > > > > > Regards,
> > > > > > Saikat
> > > > > >
> > > > > > On Sun, Oct 13, 2019 at 6:20 PM Emmanouil Gkatziouras <
> > > > > > gkatzioura@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi @Saikat Maitra
> > > > > > >
> > > > > > > I have my implementation on an Ignite fork on my personal
> GitHub
> > > > > account
> > > > > > > [1]
> > > > > > > It is based on the Kafka implementation which is of a polling
> > > nature.
> > > > > > > GCP does not provide a service like Kafka. Instead it provides
> us
> > > > with
> > > > > > > Pub/Sub which is just like Kafka.
> > > > > > > Therefore using the Kafka streamer as a guideline made
more
> sense
> > > to
> > > > > me.
> > > > > > >
> > > > > > > Apart from the implementation I have also added a unit
test.
> > > > > > > It definitely needs some polishing and a double check on
my
> side
> > to
> > > > > make
> > > > > > > the review smooth.
> > > > > > > For the tests a local Pub/Sub [2] server has to be setup
> however
> > I
> > > am
> > > > > not
> > > > > > > sure which is the best way to do so with regards to your
CI/CD.
> > > > > > >
> > > > > > > [1]
> > > https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub
> > > > > > > [2] https://egkatzioura.com/2019/09/22/pub-sub-local-emulator/
> > > > > > >
> > > > > > > *Emmanouil Gkatziouras*
> > > > > > > https://egkatzioura.com/ |
> > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > > > > > > https://github.com/gkatzioura
> > > > > > >
> > > > > > >
> > > > > > > On Sat, 12 Oct 2019 at 02:11, Saikat Maitra <
> > > saikat.maitra@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello Emmanouil,
> > > > > > > >
> > > > > > > > Can you please share more info about the implementation?
> > > > > > > >
> > > > > > > > We have similar implementation for data streamers
with Kafka
> > and
> > > > > > RocketMQ
> > > > > > > >
> > > > > > > > https://apacheignite-mix.readme.io/docs/kafka-streamer
> > > > > > > >
> > > > > > > > https://apacheignite-mix.readme.io/docs/rocketmq-streamer
> > > > > > > >
> > > > > > > > Please review and share if you have any questions.
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Saikat
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Oct 11, 2019 at 4:56 PM Emmanouil Gkatziouras
<
> > > > > > > > gkatzioura@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > By mentioning Pub/Sub I mean Google Cloud's Pub/Sub
[1]
> > > > > > > > > Yes it is indeed cloud specific. It is the messaging
> service
> > > > > provided
> > > > > > > by
> > > > > > > > > GCP.
> > > > > > > > > I have made the integration with it, but I haven't
> published
> > it
> > > > yet
> > > > > > (I
> > > > > > > am
> > > > > > > > > waiting for the discussion/ticket and proceed
with the
> > > > > contribution).
> > > > > > > > >
> > > > > > > > > Kind regards,
> > > > > > > > > Emmanouil
> > > > > > > > >
> > > > > > > > > [1] https://cloud.google.com/pubsub/docs/overview
> > > > > > > > >
> > > > > > > > > *Emmanouil Gkatziouras*
> > > > > > > > > https://egkatzioura.com/ |
> > > > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > > > > > > > > https://github.com/gkatzioura
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, 11 Oct 2019 at 21:45, Denis Magda <
> dmagda@apache.org
> > >
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hello Emmanouil,
> > > > > > > > > >
> > > > > > > > > > Are you referring to any specific implementation
tailored
> > > for a
> > > > > > > cloud?
> > > > > > > > If
> > > > > > > > > > I'm not mistaken you were mentioning AWS/GCE.
Is there
> any
> > > > Github
> > > > > > > > source
> > > > > > > > > > code?
> > > > > > > > > >
> > > > > > > > > > Here is a list of existing integrations
with streaming
> and
> > > > > pub/sub
> > > > > > > > > > frameworks:
> > > > > > > > > > https://apacheignite-mix.readme.io/docs/overview
> > > > > > > > > >
> > > > > > > > > > with the source code in Ignite master:
> > > > > > > > > > https://github.com/apache/ignite/tree/master/modules
> > > > > > > > > >
> > > > > > > > > > Check out those resources to see what are
the most
> suitable
> > > > > Ignite
> > > > > > > APIs
> > > > > > > > > for
> > > > > > > > > > such integrations.
> > > > > > > > > >
> > > > > > > > > > @Saikat Maitra <saikat.maitra@gmail.com>
, @Roman <
> > > > > > rshtykh@yahoo.com
> > > > > > > >
> > > > > > > > ,
> > > > > > > > > as
> > > > > > > > > > streaming technologies contributors/maintainers,
could
> you
> > > > please
> > > > > > > join
> > > > > > > > > this
> > > > > > > > > > thread?
> > > > > > > > > >
> > > > > > > > > > -
> > > > > > > > > > Denis
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Thu, Oct 10, 2019 at 7:58 AM Emmanouil
Gkatziouras <
> > > > > > > > > > gkatzioura@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hello everyone,
> > > > > > > > > > >
> > > > > > > > > > > I started using Ignite lately. Part
of my work involves
> > > > > Pub/Sub,
> > > > > > > > thus I
> > > > > > > > > > > created a pub/sub streamer.
> > > > > > > > > > > Pub/Sub is pretty close to Kafka.
> > > > > > > > > > > Here's how the streamer works. Every
node creates an
> > > instance
> > > > > of
> > > > > > a
> > > > > > > > > > Pub/Sub
> > > > > > > > > > > client.
> > > > > > > > > > > The clients subscribe to the same topic
using the same
> > > > > > subscription
> > > > > > > > > name.
> > > > > > > > > > > A message is being sent and one of
the nodes will
> receive
> > > the
> > > > > > > > message,
> > > > > > > > > > > process it and then forward it to Ignite
using the
> > > extractor
> > > > > > > > provided.
> > > > > > > > > > >
> > > > > > > > > > > Any thoughts?
> > > > > > > > > > >
> > > > > > > > > > > *Emmanouil Gkatziouras*
> > > > > > > > > > > https://egkatzioura.com/ |
> > > > > > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > > > > > > > > > > https://github.com/gkatzioura
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message