ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ilya Kasnacheev <ilya.kasnach...@gmail.com>
Subject Re: [DISCUSS] Pub/Sub Streamer Implementation
Date Thu, 24 Oct 2019 15:59:41 GMT
Hello!

My take on this: we want to offload some of integrations, but please commit
and release them to another project first, and only then we will decide to
remove them from our repository.

Please also make sure they are available with same artifact name and API.
If this has to be changed, let's wait for AI 3.0

Regards,
-- 
Ilya Kasnacheev


ср, 23 окт. 2019 г. в 03:23, Saikat Maitra <saikat.maitra@gmail.com>:

> Hello Emmanouil,
>
> As discussed earlier, I discussed with Apache Bahir community and they are
> interested to have Apache Ignite extensions as part of Apache Bahir
> project.
>
> Can you please share Pub/Sub streamer implementation details with Apache
> Bahir community and request for feedback.
>
> https://bahir.apache.org/
>
> I have also requested for contributor access in Jira for Apache Bahir
> project so that I can create issues and assign to myself.
>
> I can help with code reviews as well.
>
> Here is the email thread for reference
> https://www.mail-archive.com/dev@bahir.apache.org/msg02703.html
>
> Regards,
> Saikat
>
>
>
>
>
>
>
> On Mon, Oct 21, 2019 at 7:54 PM Saikat Maitra <saikat.maitra@gmail.com>
> wrote:
>
> > Hello,
> >
> > As discussed I have created following discussion threads on our migration
> > proposals for Apache Ignite extensions:
> >
> >
> >
> http://apache-ignite-users.70518.x6.nabble.com/DISCUSS-Proposal-for-Ignite-Extensions-as-a-separate-Bahir-module-or-Incubator-project-td29829.html
> >
> > https://www.mail-archive.com/dev@bahir.apache.org/msg02703.html
> >
> > I will share update as I hear more information.
> >
> > Regards,
> > Saikat
> >
> >
> >
> > On Fri, Oct 18, 2019 at 9:03 PM Saikat Maitra <saikat.maitra@gmail.com>
> > wrote:
> >
> >> Hello Denis,
> >>
> >> Sure, sounds good. I will create a separate discussion thread to get
> >> community feedback on whether we
> >> should join the Bahir or kick-off "Ignite Extensions" project.
> >>
> >> Regards,
> >> Saikat
> >>
> >> On Thu, Oct 17, 2019 at 2:21 PM Denis Magda <dmagda@apache.org> wrote:
> >>
> >>> Folks,
> >>>
> >>> The concept of Apache Bahir is precisely what we need! Saikat, thanks
> for
> >>> doing the research. Why I believe the * idea* fits us well:
> >>>
> >>>    - All integrations can be stored in separate Github repositories and
> >>>    have their dev lifecycles. We've not obliged to couple all the
> >>> integrations
> >>>    in a single repo. For instance, Spark can be located in a dedicated
> >>> repo
> >>>    while streaming integrations might be in a single one. It's up to
> us.
> >>>    - All the repositories and integrations belong to ASF. We're not
> >>> dumping
> >>>    them on Github but rather keep supporting and developing in
> >>> accordance with
> >>>    ASF vision and practices.
> >>>    - There is a way to reward contributors via committership and PMC
> >>>    membership. You won't get this if a project is just one of the
> >>> millions of
> >>>    Github projects.
> >>>
> >>> Saikat, as Ignite PMC member, could you please kick-off a separate
> >>> dev-list
> >>> discussion to involve more community members and referring to this
> >>> thread?
> >>> I think the primary question the community needs to answer whether we
> >>> should join the Bahir or kick-off "Ignite Extensions" project?
> >>>
> >>> -
> >>> Denis
> >>>
> >>>
> >>> On Thu, Oct 17, 2019 at 2:54 AM Alexey Zinoviev <
> zaleslaw.sin@gmail.com>
> >>> wrote:
> >>>
> >>> > Maybe we could move all our Streaming Integrations there, but what
is
> >>> about
> >>> > maintaining and committer permissions to the new repositories?
> >>> > I see the list of the committers and PMC members there
> >>> > https://bahir.apache.org/community-members/
> >>> >
> >>> > Could somebody from Ignite community be added to this list as a
> >>> > PMC/committer to maintain Ignite integrations?
> >>> > If yes, I'd happy to join this project and collaborate with these
> guys
> >>> > together
> >>> > If no, and we should start from the zero level with external PRs -
> >>> hmmm,
> >>> > it's better to have our own external repository with ApacheBahirr
> >>> ideology.
> >>> >
> >>> > Also, I agree, that all connectors could be moved there (in
> >>> ApacheBahirr
> >>> > like repository), but not all modules from the page
> >>> >
> >>> >
> >>>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> >>> > because
> >>> > they use different parts of core modules and could be developed with
> >>> > different velocity.
> >>> >
> >>> > In reality, before creation of new repositories we need wide
> discussion
> >>> > based on the document mentioned above.
> >>> >
> >>> > P.S. Of course, we could experiment in the next release 2.8 with one
> >>> or two
> >>> > integrations like twitter or ZeroMQ
> >>> > Also, I'd like Denis Magda idea for the separate repositories in
> Apache
> >>> > Github to maintain them by anybody.
> >>> >
> >>> >
> >>> > чт, 17 окт. 2019 г. в 05:02, Saikat Maitra <saikat.maitra@gmail.com
> >:
> >>> >
> >>> > > Hi Denis, Emmanouil
> >>> > >
> >>> > > Thank you for your email. I think creating a separate integration
> >>> project
> >>> > > in github and also proposing it as an apache incubator project
will
> >>> be a
> >>> > > good move. The other separate integration modules
> >>> > >
> >>> > >
> >>> >
> >>>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> >>> > > can
> >>> > > be moved to this new apache incubator project.
> >>> > >
> >>> > > There are similar integration available for Flink and Spark in
> Apache
> >>> > Bahir
> >>> > > https://bahir.apache.org/
> >>> > >
> >>> > > Do you think we should reach out to Apache Bahir community with
a
> >>> > proposal
> >>> > > or we should plan to create a separate Apache Incubator project?
> >>> > >
> >>> > > Regards,
> >>> > > Saikat
> >>> > >
> >>> > >
> >>> > >
> >>> > > On Wed, Oct 16, 2019 at 6:21 AM Emmanouil Gkatziouras <
> >>> > > gkatzioura@gmail.com>
> >>> > > wrote:
> >>> > >
> >>> > > > Hi all!
> >>> > > >
> >>> > > > Based on the discussions most probably this is going to be
on
> >>> another
> >>> > > > GitHub repo. Therefore I will prepare a standalone project
with
> the
> >>> > > Pub/Sub
> >>> > > > module and once the repository is created a pull request
shall be
> >>> > issued.
> >>> > > > If there is anything else I can do in order to facilitate
this
> >>> please
> >>> > let
> >>> > > > me know.
> >>> > > >
> >>> > > > If a ticket is created for this issue, I shall be able to
create
> a
> >>> pull
> >>> > > > request and thus a build will take place on ignite's CI.
> >>> > > > If everyone is aligned with adding a Pub/Sub implementation
may I
> >>> > create
> >>> > > a
> >>> > > > ticket on JIRA?
> >>> > > >
> >>> > > > My next question has to do on how Ignite configures the build
> jobs
> >>> in
> >>> > > Team
> >>> > > > City. The unit tests for my implementation need a local Pub/Sub
> >>> server
> >>> > up
> >>> > > > and running (they provide one for testing purposes). Will
this
> take
> >>> > > effect
> >>> > > > in a form of a build script inside my implementation or is
it
> >>> something
> >>> > > > that should be configured on Team City?
> >>> > > >
> >>> > > > Kind regards,
> >>> > > > Emmanouil
> >>> > > >
> >>> > > >
> >>> > > > *Emmanouil Gkatziouras*
> >>> > > > https://egkatzioura.com/ |
> >>> > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> >>> > > > https://github.com/gkatzioura
> >>> > > >
> >>> > > >
> >>> > > > On Tue, 15 Oct 2019 at 23:27, Denis Magda <dmagda@apache.org>
> >>> wrote:
> >>> > > >
> >>> > > > > Emmanouil, Saikat,
> >>> > > > >
> >>> > > > > After contemplating on this, let me propose another
way with
> this
> >>> > > > > particular integration that is aligned with our modularization
> >>> > > endeavor.
> >>> > > > > The modularization [1] defines the Ignite core, Ignite
modules,
> >>> and
> >>> > > > > independent integrations. The first two are to be located
in
> >>> Ignite
> >>> > > > > repositories and to be tested/updated by the community
for
> every
> >>> > > release.
> >>> > > > > The modules are a subset of existing integrations that
are
> >>> adopted
> >>> > > widely
> >>> > > > > and require community attention.
> >>> > > > >
> >>> > > > > As for the independent integrations [1], those are supposed
to
> be
> >>> > > located
> >>> > > > > in a separate GitHub repo. I would advise us to start
> >>> implementing
> >>> > this
> >>> > > > > concept with the Pub/Sub integration. Even though it
will be
> >>> located
> >>> > > in a
> >>> > > > > separate Github repo, it will be listed among officially
> >>> available
> >>> > > > > integrations on the Ignite website featuring Emmanouil
as a
> >>> > > contributor.
> >>> > > > >
> >>> > > > > What do you think?
> >>> > > > >
> >>> > > > >
> >>> > > > > [1]
> >>> > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> >>> > > > >
> >>> > > > >
> >>> > > > >
> >>> > > > > -
> >>> > > > > Denis
> >>> > > > >
> >>> > > > >
> >>> > > > > On Mon, Oct 14, 2019 at 9:47 PM Saikat Maitra <
> >>> > saikat.maitra@gmail.com
> >>> > > >
> >>> > > > > wrote:
> >>> > > > >
> >>> > > > > > Hi Emmanouil,
> >>> > > > > >
> >>> > > > > > The changes looks good to me. I wanted to check
if you are
> also
> >>> > able
> >>> > > to
> >>> > > > > > validate that the once data is added to Ignite
cluster you
> are
> >>> also
> >>> > > > able
> >>> > > > > to
> >>> > > > > > access it using another Ignite client or using
rest
> endpoints.
> >>> > > > > >
> >>> > > > > > We use Teamcity for CI process, the details are
as follows
> >>> > > > > >
> >>> > > > > > Trigger validation of those test suites that have
been
> >>> affected by
> >>> > > your
> >>> > > > > > changes on TeamCity <http://ci.ignite.apache.org/>:
<
> >>> > > > > http://204.14.53.153/
> >>> > > > > > >
> >>> > > > > >
> >>> > > > > >    - Locate a test suite you have to check, press
button
> named
> >>> > "..."
> >>> > > > that
> >>> > > > > >    is located on the left of "Run" button. "Run
custom build"
> >>> > dialog
> >>> > > > will
> >>> > > > > >    appear;
> >>> > > > > >    - Go to "Changes" tab and choose
> >>> > "pull/<pull-request-number>/head"
> >>> > > > in
> >>> > > > > >    "Build branch" dropdown list;
> >>> > > > > >    - Press "Run build" button and monitor tests
results.
> >>> > > > > >
> >>> > > > > >
> >>> > > > > >
> >>> > > > > > @Roman rshtykh@yahoo.com - can you also please
take a look?
> >>> > > > > >
> >>> > > > > > Regards,
> >>> > > > > > Saikat
> >>> > > > > >
> >>> > > > > > On Sun, Oct 13, 2019 at 6:20 PM Emmanouil Gkatziouras
<
> >>> > > > > > gkatzioura@gmail.com>
> >>> > > > > > wrote:
> >>> > > > > >
> >>> > > > > > > Hi @Saikat Maitra
> >>> > > > > > >
> >>> > > > > > > I have my implementation on an Ignite fork
on my personal
> >>> GitHub
> >>> > > > > account
> >>> > > > > > > [1]
> >>> > > > > > > It is based on the Kafka implementation which
is of a
> polling
> >>> > > nature.
> >>> > > > > > > GCP does not provide a service like Kafka.
Instead it
> >>> provides us
> >>> > > > with
> >>> > > > > > > Pub/Sub which is just like Kafka.
> >>> > > > > > > Therefore using the Kafka streamer as a guideline
made more
> >>> sense
> >>> > > to
> >>> > > > > me.
> >>> > > > > > >
> >>> > > > > > > Apart from the implementation I have also
added a unit
> test.
> >>> > > > > > > It definitely needs some polishing and a double
check on my
> >>> side
> >>> > to
> >>> > > > > make
> >>> > > > > > > the review smooth.
> >>> > > > > > > For the tests a local Pub/Sub [2] server has
to be setup
> >>> however
> >>> > I
> >>> > > am
> >>> > > > > not
> >>> > > > > > > sure which is the best way to do so with regards
to your
> >>> CI/CD.
> >>> > > > > > >
> >>> > > > > > > [1]
> >>> > > https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub
> >>> > > > > > > [2]
> >>> https://egkatzioura.com/2019/09/22/pub-sub-local-emulator/
> >>> > > > > > >
> >>> > > > > > > *Emmanouil Gkatziouras*
> >>> > > > > > > https://egkatzioura.com/ |
> >>> > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> >>> > > > > > > https://github.com/gkatzioura
> >>> > > > > > >
> >>> > > > > > >
> >>> > > > > > > On Sat, 12 Oct 2019 at 02:11, Saikat Maitra
<
> >>> > > saikat.maitra@gmail.com
> >>> > > > >
> >>> > > > > > > wrote:
> >>> > > > > > >
> >>> > > > > > > > Hello Emmanouil,
> >>> > > > > > > >
> >>> > > > > > > > Can you please share more info about
the implementation?
> >>> > > > > > > >
> >>> > > > > > > > We have similar implementation for data
streamers with
> >>> Kafka
> >>> > and
> >>> > > > > > RocketMQ
> >>> > > > > > > >
> >>> > > > > > > > https://apacheignite-mix.readme.io/docs/kafka-streamer
> >>> > > > > > > >
> >>> > > > > > > >
> https://apacheignite-mix.readme.io/docs/rocketmq-streamer
> >>> > > > > > > >
> >>> > > > > > > > Please review and share if you have any
questions.
> >>> > > > > > > >
> >>> > > > > > > > Regards,
> >>> > > > > > > > Saikat
> >>> > > > > > > >
> >>> > > > > > > >
> >>> > > > > > > >
> >>> > > > > > > > On Fri, Oct 11, 2019 at 4:56 PM Emmanouil
Gkatziouras <
> >>> > > > > > > > gkatzioura@gmail.com>
> >>> > > > > > > > wrote:
> >>> > > > > > > >
> >>> > > > > > > > > Hi all,
> >>> > > > > > > > >
> >>> > > > > > > > > By mentioning Pub/Sub I mean Google
Cloud's Pub/Sub [1]
> >>> > > > > > > > > Yes it is indeed cloud specific.
It is the messaging
> >>> service
> >>> > > > > provided
> >>> > > > > > > by
> >>> > > > > > > > > GCP.
> >>> > > > > > > > > I have made the integration with
it, but I haven't
> >>> published
> >>> > it
> >>> > > > yet
> >>> > > > > > (I
> >>> > > > > > > am
> >>> > > > > > > > > waiting for the discussion/ticket
and proceed with the
> >>> > > > > contribution).
> >>> > > > > > > > >
> >>> > > > > > > > > Kind regards,
> >>> > > > > > > > > Emmanouil
> >>> > > > > > > > >
> >>> > > > > > > > > [1] https://cloud.google.com/pubsub/docs/overview
> >>> > > > > > > > >
> >>> > > > > > > > > *Emmanouil Gkatziouras*
> >>> > > > > > > > > https://egkatzioura.com/ |
> >>> > > > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> >>> > > > > > > > > https://github.com/gkatzioura
> >>> > > > > > > > >
> >>> > > > > > > > >
> >>> > > > > > > > > On Fri, 11 Oct 2019 at 21:45, Denis
Magda <
> >>> dmagda@apache.org
> >>> > >
> >>> > > > > wrote:
> >>> > > > > > > > >
> >>> > > > > > > > > > Hello Emmanouil,
> >>> > > > > > > > > >
> >>> > > > > > > > > > Are you referring to any specific
implementation
> >>> tailored
> >>> > > for a
> >>> > > > > > > cloud?
> >>> > > > > > > > If
> >>> > > > > > > > > > I'm not mistaken you were mentioning
AWS/GCE. Is
> there
> >>> any
> >>> > > > Github
> >>> > > > > > > > source
> >>> > > > > > > > > > code?
> >>> > > > > > > > > >
> >>> > > > > > > > > > Here is a list of existing
integrations with
> streaming
> >>> and
> >>> > > > > pub/sub
> >>> > > > > > > > > > frameworks:
> >>> > > > > > > > > > https://apacheignite-mix.readme.io/docs/overview
> >>> > > > > > > > > >
> >>> > > > > > > > > > with the source code in Ignite
master:
> >>> > > > > > > > > > https://github.com/apache/ignite/tree/master/modules
> >>> > > > > > > > > >
> >>> > > > > > > > > > Check out those resources to
see what are the most
> >>> suitable
> >>> > > > > Ignite
> >>> > > > > > > APIs
> >>> > > > > > > > > for
> >>> > > > > > > > > > such integrations.
> >>> > > > > > > > > >
> >>> > > > > > > > > > @Saikat Maitra <saikat.maitra@gmail.com>
, @Roman <
> >>> > > > > > rshtykh@yahoo.com
> >>> > > > > > > >
> >>> > > > > > > > ,
> >>> > > > > > > > > as
> >>> > > > > > > > > > streaming technologies contributors/maintainers,
> could
> >>> you
> >>> > > > please
> >>> > > > > > > join
> >>> > > > > > > > > this
> >>> > > > > > > > > > thread?
> >>> > > > > > > > > >
> >>> > > > > > > > > > -
> >>> > > > > > > > > > Denis
> >>> > > > > > > > > >
> >>> > > > > > > > > >
> >>> > > > > > > > > > On Thu, Oct 10, 2019 at 7:58
AM Emmanouil
> Gkatziouras <
> >>> > > > > > > > > > gkatzioura@gmail.com>
> >>> > > > > > > > > > wrote:
> >>> > > > > > > > > >
> >>> > > > > > > > > > > Hello everyone,
> >>> > > > > > > > > > >
> >>> > > > > > > > > > > I started using Ignite
lately. Part of my work
> >>> involves
> >>> > > > > Pub/Sub,
> >>> > > > > > > > thus I
> >>> > > > > > > > > > > created a pub/sub streamer.
> >>> > > > > > > > > > > Pub/Sub is pretty close
to Kafka.
> >>> > > > > > > > > > > Here's how the streamer
works. Every node creates
> an
> >>> > > instance
> >>> > > > > of
> >>> > > > > > a
> >>> > > > > > > > > > Pub/Sub
> >>> > > > > > > > > > > client.
> >>> > > > > > > > > > > The clients subscribe
to the same topic using the
> >>> same
> >>> > > > > > subscription
> >>> > > > > > > > > name.
> >>> > > > > > > > > > > A message is being sent
and one of the nodes will
> >>> receive
> >>> > > the
> >>> > > > > > > > message,
> >>> > > > > > > > > > > process it and then forward
it to Ignite using the
> >>> > > extractor
> >>> > > > > > > > provided.
> >>> > > > > > > > > > >
> >>> > > > > > > > > > > Any thoughts?
> >>> > > > > > > > > > >
> >>> > > > > > > > > > > *Emmanouil Gkatziouras*
> >>> > > > > > > > > > > https://egkatzioura.com/
|
> >>> > > > > > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
> >>> > > > > > > > > > > https://github.com/gkatzioura
> >>> > > > > > > > > > >
> >>> > > > > > > > > >
> >>> > > > > > > > >
> >>> > > > > > > >
> >>> > > > > > >
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message