ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saikat Maitra <saikat.mai...@gmail.com>
Subject Re: [DISCUSS] Pub/Sub Streamer Implementation
Date Fri, 25 Oct 2019 02:35:05 GMT
Hello Ilya,

Thank you for your email. Yes, before we remove any integration we will
move them to another repository and release.

We are thinking we will start with Pub/Sub Streamer implementation and
continue to move other integration as mentioned in the following doc.

https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations

Regards,
Saikat


On Thu, Oct 24, 2019 at 11:00 AM Ilya Kasnacheev <ilya.kasnacheev@gmail.com>
wrote:

> Hello!
>
> My take on this: we want to offload some of integrations, but please commit
> and release them to another project first, and only then we will decide to
> remove them from our repository.
>
> Please also make sure they are available with same artifact name and API.
> If this has to be changed, let's wait for AI 3.0
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 23 окт. 2019 г. в 03:23, Saikat Maitra <saikat.maitra@gmail.com>:
>
> > Hello Emmanouil,
> >
> > As discussed earlier, I discussed with Apache Bahir community and they
> are
> > interested to have Apache Ignite extensions as part of Apache Bahir
> > project.
> >
> > Can you please share Pub/Sub streamer implementation details with Apache
> > Bahir community and request for feedback.
> >
> > https://bahir.apache.org/
> >
> > I have also requested for contributor access in Jira for Apache Bahir
> > project so that I can create issues and assign to myself.
> >
> > I can help with code reviews as well.
> >
> > Here is the email thread for reference
> > https://www.mail-archive.com/dev@bahir.apache.org/msg02703.html
> >
> > Regards,
> > Saikat
> >
> >
> >
> >
> >
> >
> >
> > On Mon, Oct 21, 2019 at 7:54 PM Saikat Maitra <saikat.maitra@gmail.com>
> > wrote:
> >
> > > Hello,
> > >
> > > As discussed I have created following discussion threads on our
> migration
> > > proposals for Apache Ignite extensions:
> > >
> > >
> > >
> >
> http://apache-ignite-users.70518.x6.nabble.com/DISCUSS-Proposal-for-Ignite-Extensions-as-a-separate-Bahir-module-or-Incubator-project-td29829.html
> > >
> > > https://www.mail-archive.com/dev@bahir.apache.org/msg02703.html
> > >
> > > I will share update as I hear more information.
> > >
> > > Regards,
> > > Saikat
> > >
> > >
> > >
> > > On Fri, Oct 18, 2019 at 9:03 PM Saikat Maitra <saikat.maitra@gmail.com
> >
> > > wrote:
> > >
> > >> Hello Denis,
> > >>
> > >> Sure, sounds good. I will create a separate discussion thread to get
> > >> community feedback on whether we
> > >> should join the Bahir or kick-off "Ignite Extensions" project.
> > >>
> > >> Regards,
> > >> Saikat
> > >>
> > >> On Thu, Oct 17, 2019 at 2:21 PM Denis Magda <dmagda@apache.org>
> wrote:
> > >>
> > >>> Folks,
> > >>>
> > >>> The concept of Apache Bahir is precisely what we need! Saikat, thanks
> > for
> > >>> doing the research. Why I believe the * idea* fits us well:
> > >>>
> > >>>    - All integrations can be stored in separate Github repositories
> and
> > >>>    have their dev lifecycles. We've not obliged to couple all the
> > >>> integrations
> > >>>    in a single repo. For instance, Spark can be located in a
> dedicated
> > >>> repo
> > >>>    while streaming integrations might be in a single one. It's up to
> > us.
> > >>>    - All the repositories and integrations belong to ASF. We're not
> > >>> dumping
> > >>>    them on Github but rather keep supporting and developing in
> > >>> accordance with
> > >>>    ASF vision and practices.
> > >>>    - There is a way to reward contributors via committership and PMC
> > >>>    membership. You won't get this if a project is just one of the
> > >>> millions of
> > >>>    Github projects.
> > >>>
> > >>> Saikat, as Ignite PMC member, could you please kick-off a separate
> > >>> dev-list
> > >>> discussion to involve more community members and referring to this
> > >>> thread?
> > >>> I think the primary question the community needs to answer whether
we
> > >>> should join the Bahir or kick-off "Ignite Extensions" project?
> > >>>
> > >>> -
> > >>> Denis
> > >>>
> > >>>
> > >>> On Thu, Oct 17, 2019 at 2:54 AM Alexey Zinoviev <
> > zaleslaw.sin@gmail.com>
> > >>> wrote:
> > >>>
> > >>> > Maybe we could move all our Streaming Integrations there, but
what
> is
> > >>> about
> > >>> > maintaining and committer permissions to the new repositories?
> > >>> > I see the list of the committers and PMC members there
> > >>> > https://bahir.apache.org/community-members/
> > >>> >
> > >>> > Could somebody from Ignite community be added to this list as
a
> > >>> > PMC/committer to maintain Ignite integrations?
> > >>> > If yes, I'd happy to join this project and collaborate with these
> > guys
> > >>> > together
> > >>> > If no, and we should start from the zero level with external PRs
-
> > >>> hmmm,
> > >>> > it's better to have our own external repository with ApacheBahirr
> > >>> ideology.
> > >>> >
> > >>> > Also, I agree, that all connectors could be moved there (in
> > >>> ApacheBahirr
> > >>> > like repository), but not all modules from the page
> > >>> >
> > >>> >
> > >>>
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> > >>> > because
> > >>> > they use different parts of core modules and could be developed
> with
> > >>> > different velocity.
> > >>> >
> > >>> > In reality, before creation of new repositories we need wide
> > discussion
> > >>> > based on the document mentioned above.
> > >>> >
> > >>> > P.S. Of course, we could experiment in the next release 2.8 with
> one
> > >>> or two
> > >>> > integrations like twitter or ZeroMQ
> > >>> > Also, I'd like Denis Magda idea for the separate repositories
in
> > Apache
> > >>> > Github to maintain them by anybody.
> > >>> >
> > >>> >
> > >>> > чт, 17 окт. 2019 г. в 05:02, Saikat Maitra <
> saikat.maitra@gmail.com
> > >:
> > >>> >
> > >>> > > Hi Denis, Emmanouil
> > >>> > >
> > >>> > > Thank you for your email. I think creating a separate integration
> > >>> project
> > >>> > > in github and also proposing it as an apache incubator project
> will
> > >>> be a
> > >>> > > good move. The other separate integration modules
> > >>> > >
> > >>> > >
> > >>> >
> > >>>
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> > >>> > > can
> > >>> > > be moved to this new apache incubator project.
> > >>> > >
> > >>> > > There are similar integration available for Flink and Spark
in
> > Apache
> > >>> > Bahir
> > >>> > > https://bahir.apache.org/
> > >>> > >
> > >>> > > Do you think we should reach out to Apache Bahir community
with a
> > >>> > proposal
> > >>> > > or we should plan to create a separate Apache Incubator project?
> > >>> > >
> > >>> > > Regards,
> > >>> > > Saikat
> > >>> > >
> > >>> > >
> > >>> > >
> > >>> > > On Wed, Oct 16, 2019 at 6:21 AM Emmanouil Gkatziouras <
> > >>> > > gkatzioura@gmail.com>
> > >>> > > wrote:
> > >>> > >
> > >>> > > > Hi all!
> > >>> > > >
> > >>> > > > Based on the discussions most probably this is going
to be on
> > >>> another
> > >>> > > > GitHub repo. Therefore I will prepare a standalone project
with
> > the
> > >>> > > Pub/Sub
> > >>> > > > module and once the repository is created a pull request
shall
> be
> > >>> > issued.
> > >>> > > > If there is anything else I can do in order to facilitate
this
> > >>> please
> > >>> > let
> > >>> > > > me know.
> > >>> > > >
> > >>> > > > If a ticket is created for this issue, I shall be able
to
> create
> > a
> > >>> pull
> > >>> > > > request and thus a build will take place on ignite's
CI.
> > >>> > > > If everyone is aligned with adding a Pub/Sub implementation
> may I
> > >>> > create
> > >>> > > a
> > >>> > > > ticket on JIRA?
> > >>> > > >
> > >>> > > > My next question has to do on how Ignite configures
the build
> > jobs
> > >>> in
> > >>> > > Team
> > >>> > > > City. The unit tests for my implementation need a local
Pub/Sub
> > >>> server
> > >>> > up
> > >>> > > > and running (they provide one for testing purposes).
Will this
> > take
> > >>> > > effect
> > >>> > > > in a form of a build script inside my implementation
or is it
> > >>> something
> > >>> > > > that should be configured on Team City?
> > >>> > > >
> > >>> > > > Kind regards,
> > >>> > > > Emmanouil
> > >>> > > >
> > >>> > > >
> > >>> > > > *Emmanouil Gkatziouras*
> > >>> > > > https://egkatzioura.com/ |
> > >>> > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > >>> > > > https://github.com/gkatzioura
> > >>> > > >
> > >>> > > >
> > >>> > > > On Tue, 15 Oct 2019 at 23:27, Denis Magda <dmagda@apache.org>
> > >>> wrote:
> > >>> > > >
> > >>> > > > > Emmanouil, Saikat,
> > >>> > > > >
> > >>> > > > > After contemplating on this, let me propose another
way with
> > this
> > >>> > > > > particular integration that is aligned with our
> modularization
> > >>> > > endeavor.
> > >>> > > > > The modularization [1] defines the Ignite core,
Ignite
> modules,
> > >>> and
> > >>> > > > > independent integrations. The first two are to
be located in
> > >>> Ignite
> > >>> > > > > repositories and to be tested/updated by the community
for
> > every
> > >>> > > release.
> > >>> > > > > The modules are a subset of existing integrations
that are
> > >>> adopted
> > >>> > > widely
> > >>> > > > > and require community attention.
> > >>> > > > >
> > >>> > > > > As for the independent integrations [1], those
are supposed
> to
> > be
> > >>> > > located
> > >>> > > > > in a separate GitHub repo. I would advise us to
start
> > >>> implementing
> > >>> > this
> > >>> > > > > concept with the Pub/Sub integration. Even though
it will be
> > >>> located
> > >>> > > in a
> > >>> > > > > separate Github repo, it will be listed among officially
> > >>> available
> > >>> > > > > integrations on the Ignite website featuring Emmanouil
as a
> > >>> > > contributor.
> > >>> > > > >
> > >>> > > > > What do you think?
> > >>> > > > >
> > >>> > > > >
> > >>> > > > > [1]
> > >>> > > > >
> > >>> > > > >
> > >>> > > >
> > >>> > >
> > >>> >
> > >>>
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> > >>> > > > >
> > >>> > > > >
> > >>> > > > >
> > >>> > > > > -
> > >>> > > > > Denis
> > >>> > > > >
> > >>> > > > >
> > >>> > > > > On Mon, Oct 14, 2019 at 9:47 PM Saikat Maitra <
> > >>> > saikat.maitra@gmail.com
> > >>> > > >
> > >>> > > > > wrote:
> > >>> > > > >
> > >>> > > > > > Hi Emmanouil,
> > >>> > > > > >
> > >>> > > > > > The changes looks good to me. I wanted to
check if you are
> > also
> > >>> > able
> > >>> > > to
> > >>> > > > > > validate that the once data is added to Ignite
cluster you
> > are
> > >>> also
> > >>> > > > able
> > >>> > > > > to
> > >>> > > > > > access it using another Ignite client or using
rest
> > endpoints.
> > >>> > > > > >
> > >>> > > > > > We use Teamcity for CI process, the details
are as follows
> > >>> > > > > >
> > >>> > > > > > Trigger validation of those test suites that
have been
> > >>> affected by
> > >>> > > your
> > >>> > > > > > changes on TeamCity <http://ci.ignite.apache.org/>:
<
> > >>> > > > > http://204.14.53.153/
> > >>> > > > > > >
> > >>> > > > > >
> > >>> > > > > >    - Locate a test suite you have to check,
press button
> > named
> > >>> > "..."
> > >>> > > > that
> > >>> > > > > >    is located on the left of "Run" button.
"Run custom
> build"
> > >>> > dialog
> > >>> > > > will
> > >>> > > > > >    appear;
> > >>> > > > > >    - Go to "Changes" tab and choose
> > >>> > "pull/<pull-request-number>/head"
> > >>> > > > in
> > >>> > > > > >    "Build branch" dropdown list;
> > >>> > > > > >    - Press "Run build" button and monitor
tests results.
> > >>> > > > > >
> > >>> > > > > >
> > >>> > > > > >
> > >>> > > > > > @Roman rshtykh@yahoo.com - can you also please
take a
> look?
> > >>> > > > > >
> > >>> > > > > > Regards,
> > >>> > > > > > Saikat
> > >>> > > > > >
> > >>> > > > > > On Sun, Oct 13, 2019 at 6:20 PM Emmanouil
Gkatziouras <
> > >>> > > > > > gkatzioura@gmail.com>
> > >>> > > > > > wrote:
> > >>> > > > > >
> > >>> > > > > > > Hi @Saikat Maitra
> > >>> > > > > > >
> > >>> > > > > > > I have my implementation on an Ignite
fork on my personal
> > >>> GitHub
> > >>> > > > > account
> > >>> > > > > > > [1]
> > >>> > > > > > > It is based on the Kafka implementation
which is of a
> > polling
> > >>> > > nature.
> > >>> > > > > > > GCP does not provide a service like Kafka.
Instead it
> > >>> provides us
> > >>> > > > with
> > >>> > > > > > > Pub/Sub which is just like Kafka.
> > >>> > > > > > > Therefore using the Kafka streamer as
a guideline made
> more
> > >>> sense
> > >>> > > to
> > >>> > > > > me.
> > >>> > > > > > >
> > >>> > > > > > > Apart from the implementation I have
also added a unit
> > test.
> > >>> > > > > > > It definitely needs some polishing and
a double check on
> my
> > >>> side
> > >>> > to
> > >>> > > > > make
> > >>> > > > > > > the review smooth.
> > >>> > > > > > > For the tests a local Pub/Sub [2] server
has to be setup
> > >>> however
> > >>> > I
> > >>> > > am
> > >>> > > > > not
> > >>> > > > > > > sure which is the best way to do so with
regards to your
> > >>> CI/CD.
> > >>> > > > > > >
> > >>> > > > > > > [1]
> > >>> > > https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub
> > >>> > > > > > > [2]
> > >>> https://egkatzioura.com/2019/09/22/pub-sub-local-emulator/
> > >>> > > > > > >
> > >>> > > > > > > *Emmanouil Gkatziouras*
> > >>> > > > > > > https://egkatzioura.com/ |
> > >>> > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > >>> > > > > > > https://github.com/gkatzioura
> > >>> > > > > > >
> > >>> > > > > > >
> > >>> > > > > > > On Sat, 12 Oct 2019 at 02:11, Saikat
Maitra <
> > >>> > > saikat.maitra@gmail.com
> > >>> > > > >
> > >>> > > > > > > wrote:
> > >>> > > > > > >
> > >>> > > > > > > > Hello Emmanouil,
> > >>> > > > > > > >
> > >>> > > > > > > > Can you please share more info about
the
> implementation?
> > >>> > > > > > > >
> > >>> > > > > > > > We have similar implementation for
data streamers with
> > >>> Kafka
> > >>> > and
> > >>> > > > > > RocketMQ
> > >>> > > > > > > >
> > >>> > > > > > > > https://apacheignite-mix.readme.io/docs/kafka-streamer
> > >>> > > > > > > >
> > >>> > > > > > > >
> > https://apacheignite-mix.readme.io/docs/rocketmq-streamer
> > >>> > > > > > > >
> > >>> > > > > > > > Please review and share if you have
any questions.
> > >>> > > > > > > >
> > >>> > > > > > > > Regards,
> > >>> > > > > > > > Saikat
> > >>> > > > > > > >
> > >>> > > > > > > >
> > >>> > > > > > > >
> > >>> > > > > > > > On Fri, Oct 11, 2019 at 4:56 PM
Emmanouil Gkatziouras <
> > >>> > > > > > > > gkatzioura@gmail.com>
> > >>> > > > > > > > wrote:
> > >>> > > > > > > >
> > >>> > > > > > > > > Hi all,
> > >>> > > > > > > > >
> > >>> > > > > > > > > By mentioning Pub/Sub I mean
Google Cloud's Pub/Sub
> [1]
> > >>> > > > > > > > > Yes it is indeed cloud specific.
It is the messaging
> > >>> service
> > >>> > > > > provided
> > >>> > > > > > > by
> > >>> > > > > > > > > GCP.
> > >>> > > > > > > > > I have made the integration
with it, but I haven't
> > >>> published
> > >>> > it
> > >>> > > > yet
> > >>> > > > > > (I
> > >>> > > > > > > am
> > >>> > > > > > > > > waiting for the discussion/ticket
and proceed with
> the
> > >>> > > > > contribution).
> > >>> > > > > > > > >
> > >>> > > > > > > > > Kind regards,
> > >>> > > > > > > > > Emmanouil
> > >>> > > > > > > > >
> > >>> > > > > > > > > [1] https://cloud.google.com/pubsub/docs/overview
> > >>> > > > > > > > >
> > >>> > > > > > > > > *Emmanouil Gkatziouras*
> > >>> > > > > > > > > https://egkatzioura.com/ |
> > >>> > > > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > >>> > > > > > > > > https://github.com/gkatzioura
> > >>> > > > > > > > >
> > >>> > > > > > > > >
> > >>> > > > > > > > > On Fri, 11 Oct 2019 at 21:45,
Denis Magda <
> > >>> dmagda@apache.org
> > >>> > >
> > >>> > > > > wrote:
> > >>> > > > > > > > >
> > >>> > > > > > > > > > Hello Emmanouil,
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > Are you referring to any
specific implementation
> > >>> tailored
> > >>> > > for a
> > >>> > > > > > > cloud?
> > >>> > > > > > > > If
> > >>> > > > > > > > > > I'm not mistaken you were
mentioning AWS/GCE. Is
> > there
> > >>> any
> > >>> > > > Github
> > >>> > > > > > > > source
> > >>> > > > > > > > > > code?
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > Here is a list of existing
integrations with
> > streaming
> > >>> and
> > >>> > > > > pub/sub
> > >>> > > > > > > > > > frameworks:
> > >>> > > > > > > > > > https://apacheignite-mix.readme.io/docs/overview
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > with the source code in
Ignite master:
> > >>> > > > > > > > > >
> https://github.com/apache/ignite/tree/master/modules
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > Check out those resources
to see what are the most
> > >>> suitable
> > >>> > > > > Ignite
> > >>> > > > > > > APIs
> > >>> > > > > > > > > for
> > >>> > > > > > > > > > such integrations.
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > @Saikat Maitra <saikat.maitra@gmail.com>
, @Roman
> <
> > >>> > > > > > rshtykh@yahoo.com
> > >>> > > > > > > >
> > >>> > > > > > > > ,
> > >>> > > > > > > > > as
> > >>> > > > > > > > > > streaming technologies
contributors/maintainers,
> > could
> > >>> you
> > >>> > > > please
> > >>> > > > > > > join
> > >>> > > > > > > > > this
> > >>> > > > > > > > > > thread?
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > -
> > >>> > > > > > > > > > Denis
> > >>> > > > > > > > > >
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > On Thu, Oct 10, 2019 at
7:58 AM Emmanouil
> > Gkatziouras <
> > >>> > > > > > > > > > gkatzioura@gmail.com>
> > >>> > > > > > > > > > wrote:
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > > Hello everyone,
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > > I started using Ignite
lately. Part of my work
> > >>> involves
> > >>> > > > > Pub/Sub,
> > >>> > > > > > > > thus I
> > >>> > > > > > > > > > > created a pub/sub
streamer.
> > >>> > > > > > > > > > > Pub/Sub is pretty
close to Kafka.
> > >>> > > > > > > > > > > Here's how the streamer
works. Every node creates
> > an
> > >>> > > instance
> > >>> > > > > of
> > >>> > > > > > a
> > >>> > > > > > > > > > Pub/Sub
> > >>> > > > > > > > > > > client.
> > >>> > > > > > > > > > > The clients subscribe
to the same topic using the
> > >>> same
> > >>> > > > > > subscription
> > >>> > > > > > > > > name.
> > >>> > > > > > > > > > > A message is being
sent and one of the nodes will
> > >>> receive
> > >>> > > the
> > >>> > > > > > > > message,
> > >>> > > > > > > > > > > process it and then
forward it to Ignite using
> the
> > >>> > > extractor
> > >>> > > > > > > > provided.
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > > Any thoughts?
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > > *Emmanouil Gkatziouras*
> > >>> > > > > > > > > > > https://egkatzioura.com/
|
> > >>> > > > > > > > > > >
> https://www.linkedin.com/in/gkatziourasemmanouil/
> > >>> > > > > > > > > > > https://github.com/gkatzioura
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > >
> > >>> > > > > > > > >
> > >>> > > > > > > >
> > >>> > > > > > >
> > >>> > > > > >
> > >>> > > > >
> > >>> > > >
> > >>> > >
> > >>> >
> > >>>
> > >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message