ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saikat Maitra <saikat.mai...@gmail.com>
Subject Re: [DISCUSS] Pub/Sub Streamer Implementation
Date Sat, 23 Nov 2019 15:52:06 GMT
Hi Emmanouil,

Thank you for your email. Yes, the plan is once the Flink PR is merged it
will provide the Licence, parent POMs etc and you can rebase from master
branch and apply the changes on top of it.

I am getting some issues with test failures in local with
GridTestProperties as the test.properties is not present in this new
project but should be available from dependencies.

Once I address this issue, I will go ahead and merge the changes and then
we can take it from there.

Regards,
Saikat

On Fri, Nov 22, 2019 at 5:23 PM Emmanouil Gkatziouras <gkatzioura@gmail.com>
wrote:

> Hi all,
>
> I made my first pull request [1]. Since this project is brand new (no
> parent poms, licensing), I reckoned it was best to use Saikat's branch on
> Flink.
> I suppose the Flink branch will be merged first. If not please give me
> guidelines on how I should proceed next.
>
> Kind regards
> Emmanouil
>
> [1] https://github.com/apache/ignite-extensions/pull/2
>
> *Emmanouil Gkatziouras*
> https://egkatzioura.com/ |
> https://www.linkedin.com/in/gkatziourasemmanouil/
> https://github.com/gkatzioura
>
>
> On Fri, 22 Nov 2019 at 20:55, Denis Magda <dmagda@apache.org> wrote:
>
> > Awesome, ping us whenever you're ready!
> >
> > -
> > Denis
> >
> >
> > On Fri, Nov 22, 2019 at 12:52 PM Emmanouil Gkatziouras <
> > gkatzioura@gmail.com>
> > wrote:
> >
> > > Hi all!
> > >
> > > I am sorry for being late on that. I was trying to refactor the test in
> > > order not to be in need of any external tools or spinning up a server.
> > > I did forked the new repo and indeed my changes there, so a pull
> request
> > is
> > > a matter of time!
> > >
> > > Kind regards
> > >
> > > *Emmanouil Gkatziouras*
> > > https://egkatzioura.com/ |
> > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > > https://github.com/gkatzioura
> > >
> > >
> > > On Fri, 22 Nov 2019 at 20:45, Denis Magda <dmagda@apache.org> wrote:
> > >
> > > > Hi Emmanouil,
> > > >
> > > > Do you have any questions or need any support from the community?
> > > >
> > > > -
> > > > Denis
> > > >
> > > >
> > > > On Sun, Nov 10, 2019 at 3:07 PM Saikat Maitra <
> saikat.maitra@gmail.com
> > >
> > > > wrote:
> > > >
> > > >> Hi Emmanouil,
> > > >>
> > > >> Can you please take a looks at dev utils, if this is something you
> are
> > > >> looking for
> > > >> https://github.com/apache/ignite/tree/master/modules/dev-utils?
> > > >>
> > > >> IMO, if you can release Pub/Sub server in maven and then use it as
> > > >> dependency, that would be great.
> > > >>
> > > >> Regards,
> > > >> Saikat
> > > >>
> > > >> On Sun, Nov 10, 2019 at 5:00 PM Saikat Maitra <
> > saikat.maitra@gmail.com>
> > > >> wrote:
> > > >>
> > > >> > Hi Emmanouil,
> > > >> >
> > > >> > The master branch has been initialized. I have raised a PR to
> > migrate
> > > >> > Flink module to ignite-extensions.
> > > >> >
> > > >> > PR https://github.com/apache/ignite-extensions/pull/1
> > > >> > Jira : https://issues.apache.org/jira/browse/IGNITE-12356
> > > >> >
> > > >> > Regards,
> > > >> > Saikat
> > > >> >
> > > >> >
> > > >> > On Sun, Nov 10, 2019 at 3:39 PM Emmanouil Gkatziouras <
> > > >> > gkatzioura@gmail.com> wrote:
> > > >> >
> > > >> >> Hi all!
> > > >> >>
> > > >> >> This is an update. The integration is tested with mocks based on
> > the
> > > >> >> original Ignite repo [1].
> > > >> >> I am currently waiting for the first commits on the new
> integration
> > > >> >> project
> > > >> >> to take place in order to create my pull request accordingly to
> the
> > > >> >> initial
> > > >> >> commits.
> > > >> >> In order to simulate the Pub/Sub server in a test I used a test
> > > server
> > > >> >> implementation [2]. Unfortunately they don't have this util
> > uploaded
> > > >> in a
> > > >> >> maven repo, thus it needs to be hosted on a maven repo (Apache
> > > License
> > > >> >> 2.0). Is there a repo for utilities that Ignite uses for the
> > > >> test/build?
> > > >> >> I have also created a ticket [3]
> > > >> >>
> > > >> >> Kind regards
> > > >> >>
> > > >> >> [1]
> > https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub
> > > >> >> [2] https://github.com/GoogleCloudPlatform/kafka-pubsub-emulator
> > > >> >> [3] https://issues.apache.org/jira/browse/IGNITE-12262
> > > >> >>
> > > >> >> *Emmanouil Gkatziouras*
> > > >> >> https://egkatzioura.com/ |
> > > >> >> https://www.linkedin.com/in/gkatziourasemmanouil/
> > > >> >> https://github.com/gkatzioura
> > > >> >>
> > > >> >>
> > > >> >> On Fri, 25 Oct 2019 at 03:35, Saikat Maitra <
> > saikat.maitra@gmail.com
> > > >
> > > >> >> wrote:
> > > >> >>
> > > >> >> > Hello Ilya,
> > > >> >> >
> > > >> >> > Thank you for your email. Yes, before we remove any integration
> > we
> > > >> will
> > > >> >> > move them to another repository and release.
> > > >> >> >
> > > >> >> > We are thinking we will start with Pub/Sub Streamer
> > implementation
> > > >> and
> > > >> >> > continue to move other integration as mentioned in the
> following
> > > doc.
> > > >> >> >
> > > >> >> >
> > > >> >> >
> > > >> >>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> > > >> >> >
> > > >> >> > Regards,
> > > >> >> > Saikat
> > > >> >> >
> > > >> >> >
> > > >> >> > On Thu, Oct 24, 2019 at 11:00 AM Ilya Kasnacheev <
> > > >> >> > ilya.kasnacheev@gmail.com>
> > > >> >> > wrote:
> > > >> >> >
> > > >> >> > > Hello!
> > > >> >> > >
> > > >> >> > > My take on this: we want to offload some of integrations, but
> > > >> please
> > > >> >> > commit
> > > >> >> > > and release them to another project first, and only then we
> > will
> > > >> >> decide
> > > >> >> > to
> > > >> >> > > remove them from our repository.
> > > >> >> > >
> > > >> >> > > Please also make sure they are available with same artifact
> > name
> > > >> and
> > > >> >> API.
> > > >> >> > > If this has to be changed, let's wait for AI 3.0
> > > >> >> > >
> > > >> >> > > Regards,
> > > >> >> > > --
> > > >> >> > > Ilya Kasnacheev
> > > >> >> > >
> > > >> >> > >
> > > >> >> > > ср, 23 окт. 2019 г. в 03:23, Saikat Maitra <
> > > >> saikat.maitra@gmail.com>:
> > > >> >> > >
> > > >> >> > > > Hello Emmanouil,
> > > >> >> > > >
> > > >> >> > > > As discussed earlier, I discussed with Apache Bahir
> community
> > > and
> > > >> >> they
> > > >> >> > > are
> > > >> >> > > > interested to have Apache Ignite extensions as part of
> Apache
> > > >> Bahir
> > > >> >> > > > project.
> > > >> >> > > >
> > > >> >> > > > Can you please share Pub/Sub streamer implementation
> details
> > > with
> > > >> >> > Apache
> > > >> >> > > > Bahir community and request for feedback.
> > > >> >> > > >
> > > >> >> > > > https://bahir.apache.org/
> > > >> >> > > >
> > > >> >> > > > I have also requested for contributor access in Jira for
> > Apache
> > > >> >> Bahir
> > > >> >> > > > project so that I can create issues and assign to myself.
> > > >> >> > > >
> > > >> >> > > > I can help with code reviews as well.
> > > >> >> > > >
> > > >> >> > > > Here is the email thread for reference
> > > >> >> > > >
> > > https://www.mail-archive.com/dev@bahir.apache.org/msg02703.html
> > > >> >> > > >
> > > >> >> > > > Regards,
> > > >> >> > > > Saikat
> > > >> >> > > >
> > > >> >> > > >
> > > >> >> > > >
> > > >> >> > > >
> > > >> >> > > >
> > > >> >> > > >
> > > >> >> > > >
> > > >> >> > > > On Mon, Oct 21, 2019 at 7:54 PM Saikat Maitra <
> > > >> >> saikat.maitra@gmail.com
> > > >> >> > >
> > > >> >> > > > wrote:
> > > >> >> > > >
> > > >> >> > > > > Hello,
> > > >> >> > > > >
> > > >> >> > > > > As discussed I have created following discussion threads
> on
> > > our
> > > >> >> > > migration
> > > >> >> > > > > proposals for Apache Ignite extensions:
> > > >> >> > > > >
> > > >> >> > > > >
> > > >> >> > > > >
> > > >> >> > > >
> > > >> >> > >
> > > >> >> >
> > > >> >>
> > > >>
> > >
> >
> http://apache-ignite-users.70518.x6.nabble.com/DISCUSS-Proposal-for-Ignite-Extensions-as-a-separate-Bahir-module-or-Incubator-project-td29829.html
> > > >> >> > > > >
> > > >> >> > > > >
> > > >> https://www.mail-archive.com/dev@bahir.apache.org/msg02703.html
> > > >> >> > > > >
> > > >> >> > > > > I will share update as I hear more information.
> > > >> >> > > > >
> > > >> >> > > > > Regards,
> > > >> >> > > > > Saikat
> > > >> >> > > > >
> > > >> >> > > > >
> > > >> >> > > > >
> > > >> >> > > > > On Fri, Oct 18, 2019 at 9:03 PM Saikat Maitra <
> > > >> >> > saikat.maitra@gmail.com
> > > >> >> > > >
> > > >> >> > > > > wrote:
> > > >> >> > > > >
> > > >> >> > > > >> Hello Denis,
> > > >> >> > > > >>
> > > >> >> > > > >> Sure, sounds good. I will create a separate discussion
> > > thread
> > > >> to
> > > >> >> get
> > > >> >> > > > >> community feedback on whether we
> > > >> >> > > > >> should join the Bahir or kick-off "Ignite Extensions"
> > > project.
> > > >> >> > > > >>
> > > >> >> > > > >> Regards,
> > > >> >> > > > >> Saikat
> > > >> >> > > > >>
> > > >> >> > > > >> On Thu, Oct 17, 2019 at 2:21 PM Denis Magda <
> > > >> dmagda@apache.org>
> > > >> >> > > wrote:
> > > >> >> > > > >>
> > > >> >> > > > >>> Folks,
> > > >> >> > > > >>>
> > > >> >> > > > >>> The concept of Apache Bahir is precisely what we need!
> > > >> Saikat,
> > > >> >> > thanks
> > > >> >> > > > for
> > > >> >> > > > >>> doing the research. Why I believe the * idea* fits us
> > well:
> > > >> >> > > > >>>
> > > >> >> > > > >>>    - All integrations can be stored in separate Github
> > > >> >> repositories
> > > >> >> > > and
> > > >> >> > > > >>>    have their dev lifecycles. We've not obliged to
> couple
> > > all
> > > >> >> the
> > > >> >> > > > >>> integrations
> > > >> >> > > > >>>    in a single repo. For instance, Spark can be located
> > in
> > > a
> > > >> >> > > dedicated
> > > >> >> > > > >>> repo
> > > >> >> > > > >>>    while streaming integrations might be in a single
> one.
> > > >> It's
> > > >> >> up
> > > >> >> > to
> > > >> >> > > > us.
> > > >> >> > > > >>>    - All the repositories and integrations belong to
> ASF.
> > > >> We're
> > > >> >> not
> > > >> >> > > > >>> dumping
> > > >> >> > > > >>>    them on Github but rather keep supporting and
> > developing
> > > >> in
> > > >> >> > > > >>> accordance with
> > > >> >> > > > >>>    ASF vision and practices.
> > > >> >> > > > >>>    - There is a way to reward contributors via
> > > committership
> > > >> and
> > > >> >> > PMC
> > > >> >> > > > >>>    membership. You won't get this if a project is just
> > one
> > > of
> > > >> >> the
> > > >> >> > > > >>> millions of
> > > >> >> > > > >>>    Github projects.
> > > >> >> > > > >>>
> > > >> >> > > > >>> Saikat, as Ignite PMC member, could you please
> kick-off a
> > > >> >> separate
> > > >> >> > > > >>> dev-list
> > > >> >> > > > >>> discussion to involve more community members and
> > referring
> > > to
> > > >> >> this
> > > >> >> > > > >>> thread?
> > > >> >> > > > >>> I think the primary question the community needs to
> > answer
> > > >> >> whether
> > > >> >> > we
> > > >> >> > > > >>> should join the Bahir or kick-off "Ignite Extensions"
> > > >> project?
> > > >> >> > > > >>>
> > > >> >> > > > >>> -
> > > >> >> > > > >>> Denis
> > > >> >> > > > >>>
> > > >> >> > > > >>>
> > > >> >> > > > >>> On Thu, Oct 17, 2019 at 2:54 AM Alexey Zinoviev <
> > > >> >> > > > zaleslaw.sin@gmail.com>
> > > >> >> > > > >>> wrote:
> > > >> >> > > > >>>
> > > >> >> > > > >>> > Maybe we could move all our Streaming Integrations
> > there,
> > > >> but
> > > >> >> > what
> > > >> >> > > is
> > > >> >> > > > >>> about
> > > >> >> > > > >>> > maintaining and committer permissions to the new
> > > >> repositories?
> > > >> >> > > > >>> > I see the list of the committers and PMC members
> there
> > > >> >> > > > >>> > https://bahir.apache.org/community-members/
> > > >> >> > > > >>> >
> > > >> >> > > > >>> > Could somebody from Ignite community be added to this
> > > list
> > > >> as
> > > >> >> a
> > > >> >> > > > >>> > PMC/committer to maintain Ignite integrations?
> > > >> >> > > > >>> > If yes, I'd happy to join this project and
> collaborate
> > > with
> > > >> >> these
> > > >> >> > > > guys
> > > >> >> > > > >>> > together
> > > >> >> > > > >>> > If no, and we should start from the zero level with
> > > >> external
> > > >> >> PRs
> > > >> >> > -
> > > >> >> > > > >>> hmmm,
> > > >> >> > > > >>> > it's better to have our own external repository with
> > > >> >> ApacheBahirr
> > > >> >> > > > >>> ideology.
> > > >> >> > > > >>> >
> > > >> >> > > > >>> > Also, I agree, that all connectors could be moved
> there
> > > (in
> > > >> >> > > > >>> ApacheBahirr
> > > >> >> > > > >>> > like repository), but not all modules from the page
> > > >> >> > > > >>> >
> > > >> >> > > > >>> >
> > > >> >> > > > >>>
> > > >> >> > > >
> > > >> >> > >
> > > >> >> >
> > > >> >>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> > > >> >> > > > >>> > because
> > > >> >> > > > >>> > they use different parts of core modules and could be
> > > >> >> developed
> > > >> >> > > with
> > > >> >> > > > >>> > different velocity.
> > > >> >> > > > >>> >
> > > >> >> > > > >>> > In reality, before creation of new repositories we
> need
> > > >> wide
> > > >> >> > > > discussion
> > > >> >> > > > >>> > based on the document mentioned above.
> > > >> >> > > > >>> >
> > > >> >> > > > >>> > P.S. Of course, we could experiment in the next
> release
> > > 2.8
> > > >> >> with
> > > >> >> > > one
> > > >> >> > > > >>> or two
> > > >> >> > > > >>> > integrations like twitter or ZeroMQ
> > > >> >> > > > >>> > Also, I'd like Denis Magda idea for the separate
> > > >> repositories
> > > >> >> in
> > > >> >> > > > Apache
> > > >> >> > > > >>> > Github to maintain them by anybody.
> > > >> >> > > > >>> >
> > > >> >> > > > >>> >
> > > >> >> > > > >>> > чт, 17 окт. 2019 г. в 05:02, Saikat Maitra <
> > > >> >> > > saikat.maitra@gmail.com
> > > >> >> > > > >:
> > > >> >> > > > >>> >
> > > >> >> > > > >>> > > Hi Denis, Emmanouil
> > > >> >> > > > >>> > >
> > > >> >> > > > >>> > > Thank you for your email. I think creating a
> separate
> > > >> >> > integration
> > > >> >> > > > >>> project
> > > >> >> > > > >>> > > in github and also proposing it as an apache
> > incubator
> > > >> >> project
> > > >> >> > > will
> > > >> >> > > > >>> be a
> > > >> >> > > > >>> > > good move. The other separate integration modules
> > > >> >> > > > >>> > >
> > > >> >> > > > >>> > >
> > > >> >> > > > >>> >
> > > >> >> > > > >>>
> > > >> >> > > >
> > > >> >> > >
> > > >> >> >
> > > >> >>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> > > >> >> > > > >>> > > can
> > > >> >> > > > >>> > > be moved to this new apache incubator project.
> > > >> >> > > > >>> > >
> > > >> >> > > > >>> > > There are similar integration available for Flink
> and
> > > >> Spark
> > > >> >> in
> > > >> >> > > > Apache
> > > >> >> > > > >>> > Bahir
> > > >> >> > > > >>> > > https://bahir.apache.org/
> > > >> >> > > > >>> > >
> > > >> >> > > > >>> > > Do you think we should reach out to Apache Bahir
> > > >> community
> > > >> >> > with a
> > > >> >> > > > >>> > proposal
> > > >> >> > > > >>> > > or we should plan to create a separate Apache
> > Incubator
> > > >> >> > project?
> > > >> >> > > > >>> > >
> > > >> >> > > > >>> > > Regards,
> > > >> >> > > > >>> > > Saikat
> > > >> >> > > > >>> > >
> > > >> >> > > > >>> > >
> > > >> >> > > > >>> > >
> > > >> >> > > > >>> > > On Wed, Oct 16, 2019 at 6:21 AM Emmanouil
> > Gkatziouras <
> > > >> >> > > > >>> > > gkatzioura@gmail.com>
> > > >> >> > > > >>> > > wrote:
> > > >> >> > > > >>> > >
> > > >> >> > > > >>> > > > Hi all!
> > > >> >> > > > >>> > > >
> > > >> >> > > > >>> > > > Based on the discussions most probably this is
> > going
> > > to
> > > >> >> be on
> > > >> >> > > > >>> another
> > > >> >> > > > >>> > > > GitHub repo. Therefore I will prepare a
> standalone
> > > >> project
> > > >> >> > with
> > > >> >> > > > the
> > > >> >> > > > >>> > > Pub/Sub
> > > >> >> > > > >>> > > > module and once the repository is created a pull
> > > >> request
> > > >> >> > shall
> > > >> >> > > be
> > > >> >> > > > >>> > issued.
> > > >> >> > > > >>> > > > If there is anything else I can do in order to
> > > >> facilitate
> > > >> >> > this
> > > >> >> > > > >>> please
> > > >> >> > > > >>> > let
> > > >> >> > > > >>> > > > me know.
> > > >> >> > > > >>> > > >
> > > >> >> > > > >>> > > > If a ticket is created for this issue, I shall be
> > > able
> > > >> to
> > > >> >> > > create
> > > >> >> > > > a
> > > >> >> > > > >>> pull
> > > >> >> > > > >>> > > > request and thus a build will take place on
> > ignite's
> > > >> CI.
> > > >> >> > > > >>> > > > If everyone is aligned with adding a Pub/Sub
> > > >> >> implementation
> > > >> >> > > may I
> > > >> >> > > > >>> > create
> > > >> >> > > > >>> > > a
> > > >> >> > > > >>> > > > ticket on JIRA?
> > > >> >> > > > >>> > > >
> > > >> >> > > > >>> > > > My next question has to do on how Ignite
> configures
> > > the
> > > >> >> build
> > > >> >> > > > jobs
> > > >> >> > > > >>> in
> > > >> >> > > > >>> > > Team
> > > >> >> > > > >>> > > > City. The unit tests for my implementation need a
> > > local
> > > >> >> > Pub/Sub
> > > >> >> > > > >>> server
> > > >> >> > > > >>> > up
> > > >> >> > > > >>> > > > and running (they provide one for testing
> > purposes).
> > > >> Will
> > > >> >> > this
> > > >> >> > > > take
> > > >> >> > > > >>> > > effect
> > > >> >> > > > >>> > > > in a form of a build script inside my
> > implementation
> > > or
> > > >> >> is it
> > > >> >> > > > >>> something
> > > >> >> > > > >>> > > > that should be configured on Team City?
> > > >> >> > > > >>> > > >
> > > >> >> > > > >>> > > > Kind regards,
> > > >> >> > > > >>> > > > Emmanouil
> > > >> >> > > > >>> > > >
> > > >> >> > > > >>> > > >
> > > >> >> > > > >>> > > > *Emmanouil Gkatziouras*
> > > >> >> > > > >>> > > > https://egkatzioura.com/ |
> > > >> >> > > > >>> > > >
> https://www.linkedin.com/in/gkatziourasemmanouil/
> > > >> >> > > > >>> > > > https://github.com/gkatzioura
> > > >> >> > > > >>> > > >
> > > >> >> > > > >>> > > >
> > > >> >> > > > >>> > > > On Tue, 15 Oct 2019 at 23:27, Denis Magda <
> > > >> >> dmagda@apache.org
> > > >> >> > >
> > > >> >> > > > >>> wrote:
> > > >> >> > > > >>> > > >
> > > >> >> > > > >>> > > > > Emmanouil, Saikat,
> > > >> >> > > > >>> > > > >
> > > >> >> > > > >>> > > > > After contemplating on this, let me propose
> > another
> > > >> way
> > > >> >> > with
> > > >> >> > > > this
> > > >> >> > > > >>> > > > > particular integration that is aligned with our
> > > >> >> > > modularization
> > > >> >> > > > >>> > > endeavor.
> > > >> >> > > > >>> > > > > The modularization [1] defines the Ignite core,
> > > >> Ignite
> > > >> >> > > modules,
> > > >> >> > > > >>> and
> > > >> >> > > > >>> > > > > independent integrations. The first two are to
> be
> > > >> >> located
> > > >> >> > in
> > > >> >> > > > >>> Ignite
> > > >> >> > > > >>> > > > > repositories and to be tested/updated by the
> > > >> community
> > > >> >> for
> > > >> >> > > > every
> > > >> >> > > > >>> > > release.
> > > >> >> > > > >>> > > > > The modules are a subset of existing
> integrations
> > > >> that
> > > >> >> are
> > > >> >> > > > >>> adopted
> > > >> >> > > > >>> > > widely
> > > >> >> > > > >>> > > > > and require community attention.
> > > >> >> > > > >>> > > > >
> > > >> >> > > > >>> > > > > As for the independent integrations [1], those
> > are
> > > >> >> supposed
> > > >> >> > > to
> > > >> >> > > > be
> > > >> >> > > > >>> > > located
> > > >> >> > > > >>> > > > > in a separate GitHub repo. I would advise us to
> > > start
> > > >> >> > > > >>> implementing
> > > >> >> > > > >>> > this
> > > >> >> > > > >>> > > > > concept with the Pub/Sub integration. Even
> though
> > > it
> > > >> >> will
> > > >> >> > be
> > > >> >> > > > >>> located
> > > >> >> > > > >>> > > in a
> > > >> >> > > > >>> > > > > separate Github repo, it will be listed among
> > > >> officially
> > > >> >> > > > >>> available
> > > >> >> > > > >>> > > > > integrations on the Ignite website featuring
> > > >> Emmanouil
> > > >> >> as a
> > > >> >> > > > >>> > > contributor.
> > > >> >> > > > >>> > > > >
> > > >> >> > > > >>> > > > > What do you think?
> > > >> >> > > > >>> > > > >
> > > >> >> > > > >>> > > > >
> > > >> >> > > > >>> > > > > [1]
> > > >> >> > > > >>> > > > >
> > > >> >> > > > >>> > > > >
> > > >> >> > > > >>> > > >
> > > >> >> > > > >>> > >
> > > >> >> > > > >>> >
> > > >> >> > > > >>>
> > > >> >> > > >
> > > >> >> > >
> > > >> >> >
> > > >> >>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
> > > >> >> > > > >>> > > > >
> > > >> >> > > > >>> > > > >
> > > >> >> > > > >>> > > > >
> > > >> >> > > > >>> > > > > -
> > > >> >> > > > >>> > > > > Denis
> > > >> >> > > > >>> > > > >
> > > >> >> > > > >>> > > > >
> > > >> >> > > > >>> > > > > On Mon, Oct 14, 2019 at 9:47 PM Saikat Maitra <
> > > >> >> > > > >>> > saikat.maitra@gmail.com
> > > >> >> > > > >>> > > >
> > > >> >> > > > >>> > > > > wrote:
> > > >> >> > > > >>> > > > >
> > > >> >> > > > >>> > > > > > Hi Emmanouil,
> > > >> >> > > > >>> > > > > >
> > > >> >> > > > >>> > > > > > The changes looks good to me. I wanted to
> check
> > > if
> > > >> you
> > > >> >> > are
> > > >> >> > > > also
> > > >> >> > > > >>> > able
> > > >> >> > > > >>> > > to
> > > >> >> > > > >>> > > > > > validate that the once data is added to
> Ignite
> > > >> cluster
> > > >> >> > you
> > > >> >> > > > are
> > > >> >> > > > >>> also
> > > >> >> > > > >>> > > > able
> > > >> >> > > > >>> > > > > to
> > > >> >> > > > >>> > > > > > access it using another Ignite client or
> using
> > > rest
> > > >> >> > > > endpoints.
> > > >> >> > > > >>> > > > > >
> > > >> >> > > > >>> > > > > > We use Teamcity for CI process, the details
> are
> > > as
> > > >> >> > follows
> > > >> >> > > > >>> > > > > >
> > > >> >> > > > >>> > > > > > Trigger validation of those test suites that
> > have
> > > >> been
> > > >> >> > > > >>> affected by
> > > >> >> > > > >>> > > your
> > > >> >> > > > >>> > > > > > changes on TeamCity <
> > > http://ci.ignite.apache.org/>:
> > > >> <
> > > >> >> > > > >>> > > > > http://204.14.53.153/
> > > >> >> > > > >>> > > > > > >
> > > >> >> > > > >>> > > > > >
> > > >> >> > > > >>> > > > > >    - Locate a test suite you have to check,
> > press
> > > >> >> button
> > > >> >> > > > named
> > > >> >> > > > >>> > "..."
> > > >> >> > > > >>> > > > that
> > > >> >> > > > >>> > > > > >    is located on the left of "Run" button.
> "Run
> > > >> custom
> > > >> >> > > build"
> > > >> >> > > > >>> > dialog
> > > >> >> > > > >>> > > > will
> > > >> >> > > > >>> > > > > >    appear;
> > > >> >> > > > >>> > > > > >    - Go to "Changes" tab and choose
> > > >> >> > > > >>> > "pull/<pull-request-number>/head"
> > > >> >> > > > >>> > > > in
> > > >> >> > > > >>> > > > > >    "Build branch" dropdown list;
> > > >> >> > > > >>> > > > > >    - Press "Run build" button and monitor
> tests
> > > >> >> results.
> > > >> >> > > > >>> > > > > >
> > > >> >> > > > >>> > > > > >
> > > >> >> > > > >>> > > > > >
> > > >> >> > > > >>> > > > > > @Roman rshtykh@yahoo.com - can you also
> please
> > > >> take a
> > > >> >> > > look?
> > > >> >> > > > >>> > > > > >
> > > >> >> > > > >>> > > > > > Regards,
> > > >> >> > > > >>> > > > > > Saikat
> > > >> >> > > > >>> > > > > >
> > > >> >> > > > >>> > > > > > On Sun, Oct 13, 2019 at 6:20 PM Emmanouil
> > > >> Gkatziouras
> > > >> >> <
> > > >> >> > > > >>> > > > > > gkatzioura@gmail.com>
> > > >> >> > > > >>> > > > > > wrote:
> > > >> >> > > > >>> > > > > >
> > > >> >> > > > >>> > > > > > > Hi @Saikat Maitra
> > > >> >> > > > >>> > > > > > >
> > > >> >> > > > >>> > > > > > > I have my implementation on an Ignite fork
> on
> > > my
> > > >> >> > personal
> > > >> >> > > > >>> GitHub
> > > >> >> > > > >>> > > > > account
> > > >> >> > > > >>> > > > > > > [1]
> > > >> >> > > > >>> > > > > > > It is based on the Kafka implementation
> which
> > > is
> > > >> of
> > > >> >> a
> > > >> >> > > > polling
> > > >> >> > > > >>> > > nature.
> > > >> >> > > > >>> > > > > > > GCP does not provide a service like Kafka.
> > > >> Instead
> > > >> >> it
> > > >> >> > > > >>> provides us
> > > >> >> > > > >>> > > > with
> > > >> >> > > > >>> > > > > > > Pub/Sub which is just like Kafka.
> > > >> >> > > > >>> > > > > > > Therefore using the Kafka streamer as a
> > > guideline
> > > >> >> made
> > > >> >> > > more
> > > >> >> > > > >>> sense
> > > >> >> > > > >>> > > to
> > > >> >> > > > >>> > > > > me.
> > > >> >> > > > >>> > > > > > >
> > > >> >> > > > >>> > > > > > > Apart from the implementation I have also
> > > added a
> > > >> >> unit
> > > >> >> > > > test.
> > > >> >> > > > >>> > > > > > > It definitely needs some polishing and a
> > double
> > > >> >> check
> > > >> >> > on
> > > >> >> > > my
> > > >> >> > > > >>> side
> > > >> >> > > > >>> > to
> > > >> >> > > > >>> > > > > make
> > > >> >> > > > >>> > > > > > > the review smooth.
> > > >> >> > > > >>> > > > > > > For the tests a local Pub/Sub [2] server
> has
> > to
> > > >> be
> > > >> >> > setup
> > > >> >> > > > >>> however
> > > >> >> > > > >>> > I
> > > >> >> > > > >>> > > am
> > > >> >> > > > >>> > > > > not
> > > >> >> > > > >>> > > > > > > sure which is the best way to do so with
> > > regards
> > > >> to
> > > >> >> > your
> > > >> >> > > > >>> CI/CD.
> > > >> >> > > > >>> > > > > > >
> > > >> >> > > > >>> > > > > > > [1]
> > > >> >> > > > >>> > >
> > > >> >> >
> https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub
> > > >> >> > > > >>> > > > > > > [2]
> > > >> >> > > > >>>
> > https://egkatzioura.com/2019/09/22/pub-sub-local-emulator/
> > > >> >> > > > >>> > > > > > >
> > > >> >> > > > >>> > > > > > > *Emmanouil Gkatziouras*
> > > >> >> > > > >>> > > > > > > https://egkatzioura.com/ |
> > > >> >> > > > >>> > > > > > >
> > > >> https://www.linkedin.com/in/gkatziourasemmanouil/
> > > >> >> > > > >>> > > > > > > https://github.com/gkatzioura
> > > >> >> > > > >>> > > > > > >
> > > >> >> > > > >>> > > > > > >
> > > >> >> > > > >>> > > > > > > On Sat, 12 Oct 2019 at 02:11, Saikat
> Maitra <
> > > >> >> > > > >>> > > saikat.maitra@gmail.com
> > > >> >> > > > >>> > > > >
> > > >> >> > > > >>> > > > > > > wrote:
> > > >> >> > > > >>> > > > > > >
> > > >> >> > > > >>> > > > > > > > Hello Emmanouil,
> > > >> >> > > > >>> > > > > > > >
> > > >> >> > > > >>> > > > > > > > Can you please share more info about the
> > > >> >> > > implementation?
> > > >> >> > > > >>> > > > > > > >
> > > >> >> > > > >>> > > > > > > > We have similar implementation for data
> > > >> streamers
> > > >> >> > with
> > > >> >> > > > >>> Kafka
> > > >> >> > > > >>> > and
> > > >> >> > > > >>> > > > > > RocketMQ
> > > >> >> > > > >>> > > > > > > >
> > > >> >> > > > >>> > > > > > > >
> > > >> >> > https://apacheignite-mix.readme.io/docs/kafka-streamer
> > > >> >> > > > >>> > > > > > > >
> > > >> >> > > > >>> > > > > > > >
> > > >> >> > > > https://apacheignite-mix.readme.io/docs/rocketmq-streamer
> > > >> >> > > > >>> > > > > > > >
> > > >> >> > > > >>> > > > > > > > Please review and share if you have any
> > > >> questions.
> > > >> >> > > > >>> > > > > > > >
> > > >> >> > > > >>> > > > > > > > Regards,
> > > >> >> > > > >>> > > > > > > > Saikat
> > > >> >> > > > >>> > > > > > > >
> > > >> >> > > > >>> > > > > > > >
> > > >> >> > > > >>> > > > > > > >
> > > >> >> > > > >>> > > > > > > > On Fri, Oct 11, 2019 at 4:56 PM Emmanouil
> > > >> >> > Gkatziouras <
> > > >> >> > > > >>> > > > > > > > gkatzioura@gmail.com>
> > > >> >> > > > >>> > > > > > > > wrote:
> > > >> >> > > > >>> > > > > > > >
> > > >> >> > > > >>> > > > > > > > > Hi all,
> > > >> >> > > > >>> > > > > > > > >
> > > >> >> > > > >>> > > > > > > > > By mentioning Pub/Sub I mean Google
> > Cloud's
> > > >> >> Pub/Sub
> > > >> >> > > [1]
> > > >> >> > > > >>> > > > > > > > > Yes it is indeed cloud specific. It is
> > the
> > > >> >> > messaging
> > > >> >> > > > >>> service
> > > >> >> > > > >>> > > > > provided
> > > >> >> > > > >>> > > > > > > by
> > > >> >> > > > >>> > > > > > > > > GCP.
> > > >> >> > > > >>> > > > > > > > > I have made the integration with it,
> but
> > I
> > > >> >> haven't
> > > >> >> > > > >>> published
> > > >> >> > > > >>> > it
> > > >> >> > > > >>> > > > yet
> > > >> >> > > > >>> > > > > > (I
> > > >> >> > > > >>> > > > > > > am
> > > >> >> > > > >>> > > > > > > > > waiting for the discussion/ticket and
> > > proceed
> > > >> >> with
> > > >> >> > > the
> > > >> >> > > > >>> > > > > contribution).
> > > >> >> > > > >>> > > > > > > > >
> > > >> >> > > > >>> > > > > > > > > Kind regards,
> > > >> >> > > > >>> > > > > > > > > Emmanouil
> > > >> >> > > > >>> > > > > > > > >
> > > >> >> > > > >>> > > > > > > > > [1]
> > > >> >> https://cloud.google.com/pubsub/docs/overview
> > > >> >> > > > >>> > > > > > > > >
> > > >> >> > > > >>> > > > > > > > > *Emmanouil Gkatziouras*
> > > >> >> > > > >>> > > > > > > > > https://egkatzioura.com/ |
> > > >> >> > > > >>> > > > > > > > >
> > > >> >> https://www.linkedin.com/in/gkatziourasemmanouil/
> > > >> >> > > > >>> > > > > > > > > https://github.com/gkatzioura
> > > >> >> > > > >>> > > > > > > > >
> > > >> >> > > > >>> > > > > > > > >
> > > >> >> > > > >>> > > > > > > > > On Fri, 11 Oct 2019 at 21:45, Denis
> > Magda <
> > > >> >> > > > >>> dmagda@apache.org
> > > >> >> > > > >>> > >
> > > >> >> > > > >>> > > > > wrote:
> > > >> >> > > > >>> > > > > > > > >
> > > >> >> > > > >>> > > > > > > > > > Hello Emmanouil,
> > > >> >> > > > >>> > > > > > > > > >
> > > >> >> > > > >>> > > > > > > > > > Are you referring to any specific
> > > >> >> implementation
> > > >> >> > > > >>> tailored
> > > >> >> > > > >>> > > for a
> > > >> >> > > > >>> > > > > > > cloud?
> > > >> >> > > > >>> > > > > > > > If
> > > >> >> > > > >>> > > > > > > > > > I'm not mistaken you were mentioning
> > > >> AWS/GCE.
> > > >> >> Is
> > > >> >> > > > there
> > > >> >> > > > >>> any
> > > >> >> > > > >>> > > > Github
> > > >> >> > > > >>> > > > > > > > source
> > > >> >> > > > >>> > > > > > > > > > code?
> > > >> >> > > > >>> > > > > > > > > >
> > > >> >> > > > >>> > > > > > > > > > Here is a list of existing
> integrations
> > > >> with
> > > >> >> > > > streaming
> > > >> >> > > > >>> and
> > > >> >> > > > >>> > > > > pub/sub
> > > >> >> > > > >>> > > > > > > > > > frameworks:
> > > >> >> > > > >>> > > > > > > > > >
> > > >> >> https://apacheignite-mix.readme.io/docs/overview
> > > >> >> > > > >>> > > > > > > > > >
> > > >> >> > > > >>> > > > > > > > > > with the source code in Ignite
> master:
> > > >> >> > > > >>> > > > > > > > > >
> > > >> >> > > https://github.com/apache/ignite/tree/master/modules
> > > >> >> > > > >>> > > > > > > > > >
> > > >> >> > > > >>> > > > > > > > > > Check out those resources to see what
> > are
> > > >> the
> > > >> >> > most
> > > >> >> > > > >>> suitable
> > > >> >> > > > >>> > > > > Ignite
> > > >> >> > > > >>> > > > > > > APIs
> > > >> >> > > > >>> > > > > > > > > for
> > > >> >> > > > >>> > > > > > > > > > such integrations.
> > > >> >> > > > >>> > > > > > > > > >
> > > >> >> > > > >>> > > > > > > > > > @Saikat Maitra <
> > saikat.maitra@gmail.com>
> > > ,
> > > >> >> > @Roman
> > > >> >> > > <
> > > >> >> > > > >>> > > > > > rshtykh@yahoo.com
> > > >> >> > > > >>> > > > > > > >
> > > >> >> > > > >>> > > > > > > > ,
> > > >> >> > > > >>> > > > > > > > > as
> > > >> >> > > > >>> > > > > > > > > > streaming technologies
> > > >> >> contributors/maintainers,
> > > >> >> > > > could
> > > >> >> > > > >>> you
> > > >> >> > > > >>> > > > please
> > > >> >> > > > >>> > > > > > > join
> > > >> >> > > > >>> > > > > > > > > this
> > > >> >> > > > >>> > > > > > > > > > thread?
> > > >> >> > > > >>> > > > > > > > > >
> > > >> >> > > > >>> > > > > > > > > > -
> > > >> >> > > > >>> > > > > > > > > > Denis
> > > >> >> > > > >>> > > > > > > > > >
> > > >> >> > > > >>> > > > > > > > > >
> > > >> >> > > > >>> > > > > > > > > > On Thu, Oct 10, 2019 at 7:58 AM
> > Emmanouil
> > > >> >> > > > Gkatziouras <
> > > >> >> > > > >>> > > > > > > > > > gkatzioura@gmail.com>
> > > >> >> > > > >>> > > > > > > > > > wrote:
> > > >> >> > > > >>> > > > > > > > > >
> > > >> >> > > > >>> > > > > > > > > > > Hello everyone,
> > > >> >> > > > >>> > > > > > > > > > >
> > > >> >> > > > >>> > > > > > > > > > > I started using Ignite lately. Part
> > of
> > > my
> > > >> >> work
> > > >> >> > > > >>> involves
> > > >> >> > > > >>> > > > > Pub/Sub,
> > > >> >> > > > >>> > > > > > > > thus I
> > > >> >> > > > >>> > > > > > > > > > > created a pub/sub streamer.
> > > >> >> > > > >>> > > > > > > > > > > Pub/Sub is pretty close to Kafka.
> > > >> >> > > > >>> > > > > > > > > > > Here's how the streamer works.
> Every
> > > node
> > > >> >> > creates
> > > >> >> > > > an
> > > >> >> > > > >>> > > instance
> > > >> >> > > > >>> > > > > of
> > > >> >> > > > >>> > > > > > a
> > > >> >> > > > >>> > > > > > > > > > Pub/Sub
> > > >> >> > > > >>> > > > > > > > > > > client.
> > > >> >> > > > >>> > > > > > > > > > > The clients subscribe to the same
> > topic
> > > >> >> using
> > > >> >> > the
> > > >> >> > > > >>> same
> > > >> >> > > > >>> > > > > > subscription
> > > >> >> > > > >>> > > > > > > > > name.
> > > >> >> > > > >>> > > > > > > > > > > A message is being sent and one of
> > the
> > > >> nodes
> > > >> >> > will
> > > >> >> > > > >>> receive
> > > >> >> > > > >>> > > the
> > > >> >> > > > >>> > > > > > > > message,
> > > >> >> > > > >>> > > > > > > > > > > process it and then forward it to
> > > Ignite
> > > >> >> using
> > > >> >> > > the
> > > >> >> > > > >>> > > extractor
> > > >> >> > > > >>> > > > > > > > provided.
> > > >> >> > > > >>> > > > > > > > > > >
> > > >> >> > > > >>> > > > > > > > > > > Any thoughts?
> > > >> >> > > > >>> > > > > > > > > > >
> > > >> >> > > > >>> > > > > > > > > > > *Emmanouil Gkatziouras*
> > > >> >> > > > >>> > > > > > > > > > > https://egkatzioura.com/ |
> > > >> >> > > > >>> > > > > > > > > > >
> > > >> >> > > https://www.linkedin.com/in/gkatziourasemmanouil/
> > > >> >> > > > >>> > > > > > > > > > > https://github.com/gkatzioura
> > > >> >> > > > >>> > > > > > > > > > >
> > > >> >> > > > >>> > > > > > > > > >
> > > >> >> > > > >>> > > > > > > > >
> > > >> >> > > > >>> > > > > > > >
> > > >> >> > > > >>> > > > > > >
> > > >> >> > > > >>> > > > > >
> > > >> >> > > > >>> > > > >
> > > >> >> > > > >>> > > >
> > > >> >> > > > >>> > >
> > > >> >> > > > >>> >
> > > >> >> > > > >>>
> > > >> >> > > > >>
> > > >> >> > > >
> > > >> >> > >
> > > >> >> >
> > > >> >>
> > > >> >
> > > >>
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message