ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saikat Maitra <saikat.mai...@gmail.com>
Subject Re: [DISCUSS] Pub/Sub Streamer Implementation
Date Tue, 22 Oct 2019 00:54:09 GMT
Hello,

As discussed I have created following discussion threads on our migration
proposals for Apache Ignite extensions:

http://apache-ignite-users.70518.x6.nabble.com/DISCUSS-Proposal-for-Ignite-Extensions-as-a-separate-Bahir-module-or-Incubator-project-td29829.html

https://www.mail-archive.com/dev@bahir.apache.org/msg02703.html

I will share update as I hear more information.

Regards,
Saikat



On Fri, Oct 18, 2019 at 9:03 PM Saikat Maitra <saikat.maitra@gmail.com>
wrote:

> Hello Denis,
>
> Sure, sounds good. I will create a separate discussion thread to get
> community feedback on whether we
> should join the Bahir or kick-off "Ignite Extensions" project.
>
> Regards,
> Saikat
>
> On Thu, Oct 17, 2019 at 2:21 PM Denis Magda <dmagda@apache.org> wrote:
>
>> Folks,
>>
>> The concept of Apache Bahir is precisely what we need! Saikat, thanks for
>> doing the research. Why I believe the * idea* fits us well:
>>
>>    - All integrations can be stored in separate Github repositories and
>>    have their dev lifecycles. We've not obliged to couple all the
>> integrations
>>    in a single repo. For instance, Spark can be located in a dedicated
>> repo
>>    while streaming integrations might be in a single one. It's up to us.
>>    - All the repositories and integrations belong to ASF. We're not
>> dumping
>>    them on Github but rather keep supporting and developing in accordance
>> with
>>    ASF vision and practices.
>>    - There is a way to reward contributors via committership and PMC
>>    membership. You won't get this if a project is just one of the
>> millions of
>>    Github projects.
>>
>> Saikat, as Ignite PMC member, could you please kick-off a separate
>> dev-list
>> discussion to involve more community members and referring to this thread?
>> I think the primary question the community needs to answer whether we
>> should join the Bahir or kick-off "Ignite Extensions" project?
>>
>> -
>> Denis
>>
>>
>> On Thu, Oct 17, 2019 at 2:54 AM Alexey Zinoviev <zaleslaw.sin@gmail.com>
>> wrote:
>>
>> > Maybe we could move all our Streaming Integrations there, but what is
>> about
>> > maintaining and committer permissions to the new repositories?
>> > I see the list of the committers and PMC members there
>> > https://bahir.apache.org/community-members/
>> >
>> > Could somebody from Ignite community be added to this list as a
>> > PMC/committer to maintain Ignite integrations?
>> > If yes, I'd happy to join this project and collaborate with these guys
>> > together
>> > If no, and we should start from the zero level with external PRs - hmmm,
>> > it's better to have our own external repository with ApacheBahirr
>> ideology.
>> >
>> > Also, I agree, that all connectors could be moved there (in ApacheBahirr
>> > like repository), but not all modules from the page
>> >
>> >
>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
>> > because
>> > they use different parts of core modules and could be developed with
>> > different velocity.
>> >
>> > In reality, before creation of new repositories we need wide discussion
>> > based on the document mentioned above.
>> >
>> > P.S. Of course, we could experiment in the next release 2.8 with one or
>> two
>> > integrations like twitter or ZeroMQ
>> > Also, I'd like Denis Magda idea for the separate repositories in Apache
>> > Github to maintain them by anybody.
>> >
>> >
>> > чт, 17 окт. 2019 г. в 05:02, Saikat Maitra <saikat.maitra@gmail.com>:
>> >
>> > > Hi Denis, Emmanouil
>> > >
>> > > Thank you for your email. I think creating a separate integration
>> project
>> > > in github and also proposing it as an apache incubator project will
>> be a
>> > > good move. The other separate integration modules
>> > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
>> > > can
>> > > be moved to this new apache incubator project.
>> > >
>> > > There are similar integration available for Flink and Spark in Apache
>> > Bahir
>> > > https://bahir.apache.org/
>> > >
>> > > Do you think we should reach out to Apache Bahir community with a
>> > proposal
>> > > or we should plan to create a separate Apache Incubator project?
>> > >
>> > > Regards,
>> > > Saikat
>> > >
>> > >
>> > >
>> > > On Wed, Oct 16, 2019 at 6:21 AM Emmanouil Gkatziouras <
>> > > gkatzioura@gmail.com>
>> > > wrote:
>> > >
>> > > > Hi all!
>> > > >
>> > > > Based on the discussions most probably this is going to be on
>> another
>> > > > GitHub repo. Therefore I will prepare a standalone project with the
>> > > Pub/Sub
>> > > > module and once the repository is created a pull request shall be
>> > issued.
>> > > > If there is anything else I can do in order to facilitate this
>> please
>> > let
>> > > > me know.
>> > > >
>> > > > If a ticket is created for this issue, I shall be able to create a
>> pull
>> > > > request and thus a build will take place on ignite's CI.
>> > > > If everyone is aligned with adding a Pub/Sub implementation may I
>> > create
>> > > a
>> > > > ticket on JIRA?
>> > > >
>> > > > My next question has to do on how Ignite configures the build jobs
>> in
>> > > Team
>> > > > City. The unit tests for my implementation need a local Pub/Sub
>> server
>> > up
>> > > > and running (they provide one for testing purposes). Will this take
>> > > effect
>> > > > in a form of a build script inside my implementation or is it
>> something
>> > > > that should be configured on Team City?
>> > > >
>> > > > Kind regards,
>> > > > Emmanouil
>> > > >
>> > > >
>> > > > *Emmanouil Gkatziouras*
>> > > > https://egkatzioura.com/ |
>> > > > https://www.linkedin.com/in/gkatziourasemmanouil/
>> > > > https://github.com/gkatzioura
>> > > >
>> > > >
>> > > > On Tue, 15 Oct 2019 at 23:27, Denis Magda <dmagda@apache.org>
>> wrote:
>> > > >
>> > > > > Emmanouil, Saikat,
>> > > > >
>> > > > > After contemplating on this, let me propose another way with
this
>> > > > > particular integration that is aligned with our modularization
>> > > endeavor.
>> > > > > The modularization [1] defines the Ignite core, Ignite modules,
>> and
>> > > > > independent integrations. The first two are to be located in
>> Ignite
>> > > > > repositories and to be tested/updated by the community for every
>> > > release.
>> > > > > The modules are a subset of existing integrations that are adopted
>> > > widely
>> > > > > and require community attention.
>> > > > >
>> > > > > As for the independent integrations [1], those are supposed to
be
>> > > located
>> > > > > in a separate GitHub repo. I would advise us to start implementing
>> > this
>> > > > > concept with the Pub/Sub integration. Even though it will be
>> located
>> > > in a
>> > > > > separate Github repo, it will be listed among officially available
>> > > > > integrations on the Ignite website featuring Emmanouil as a
>> > > contributor.
>> > > > >
>> > > > > What do you think?
>> > > > >
>> > > > >
>> > > > > [1]
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization#IEP-36:Modularization-IndependentIntegrations
>> > > > >
>> > > > >
>> > > > >
>> > > > > -
>> > > > > Denis
>> > > > >
>> > > > >
>> > > > > On Mon, Oct 14, 2019 at 9:47 PM Saikat Maitra <
>> > saikat.maitra@gmail.com
>> > > >
>> > > > > wrote:
>> > > > >
>> > > > > > Hi Emmanouil,
>> > > > > >
>> > > > > > The changes looks good to me. I wanted to check if you are
also
>> > able
>> > > to
>> > > > > > validate that the once data is added to Ignite cluster you
are
>> also
>> > > > able
>> > > > > to
>> > > > > > access it using another Ignite client or using rest endpoints.
>> > > > > >
>> > > > > > We use Teamcity for CI process, the details are as follows
>> > > > > >
>> > > > > > Trigger validation of those test suites that have been affected
>> by
>> > > your
>> > > > > > changes on TeamCity <http://ci.ignite.apache.org/>:
<
>> > > > > http://204.14.53.153/
>> > > > > > >
>> > > > > >
>> > > > > >    - Locate a test suite you have to check, press button
named
>> > "..."
>> > > > that
>> > > > > >    is located on the left of "Run" button. "Run custom build"
>> > dialog
>> > > > will
>> > > > > >    appear;
>> > > > > >    - Go to "Changes" tab and choose
>> > "pull/<pull-request-number>/head"
>> > > > in
>> > > > > >    "Build branch" dropdown list;
>> > > > > >    - Press "Run build" button and monitor tests results.
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > @Roman rshtykh@yahoo.com - can you also please take a look?
>> > > > > >
>> > > > > > Regards,
>> > > > > > Saikat
>> > > > > >
>> > > > > > On Sun, Oct 13, 2019 at 6:20 PM Emmanouil Gkatziouras <
>> > > > > > gkatzioura@gmail.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi @Saikat Maitra
>> > > > > > >
>> > > > > > > I have my implementation on an Ignite fork on my personal
>> GitHub
>> > > > > account
>> > > > > > > [1]
>> > > > > > > It is based on the Kafka implementation which is of
a polling
>> > > nature.
>> > > > > > > GCP does not provide a service like Kafka. Instead
it
>> provides us
>> > > > with
>> > > > > > > Pub/Sub which is just like Kafka.
>> > > > > > > Therefore using the Kafka streamer as a guideline made
more
>> sense
>> > > to
>> > > > > me.
>> > > > > > >
>> > > > > > > Apart from the implementation I have also added a unit
test.
>> > > > > > > It definitely needs some polishing and a double check
on my
>> side
>> > to
>> > > > > make
>> > > > > > > the review smooth.
>> > > > > > > For the tests a local Pub/Sub [2] server has to be
setup
>> however
>> > I
>> > > am
>> > > > > not
>> > > > > > > sure which is the best way to do so with regards to
your
>> CI/CD.
>> > > > > > >
>> > > > > > > [1]
>> > > https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub
>> > > > > > > [2]
>> https://egkatzioura.com/2019/09/22/pub-sub-local-emulator/
>> > > > > > >
>> > > > > > > *Emmanouil Gkatziouras*
>> > > > > > > https://egkatzioura.com/ |
>> > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
>> > > > > > > https://github.com/gkatzioura
>> > > > > > >
>> > > > > > >
>> > > > > > > On Sat, 12 Oct 2019 at 02:11, Saikat Maitra <
>> > > saikat.maitra@gmail.com
>> > > > >
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > Hello Emmanouil,
>> > > > > > > >
>> > > > > > > > Can you please share more info about the implementation?
>> > > > > > > >
>> > > > > > > > We have similar implementation for data streamers
with Kafka
>> > and
>> > > > > > RocketMQ
>> > > > > > > >
>> > > > > > > > https://apacheignite-mix.readme.io/docs/kafka-streamer
>> > > > > > > >
>> > > > > > > > https://apacheignite-mix.readme.io/docs/rocketmq-streamer
>> > > > > > > >
>> > > > > > > > Please review and share if you have any questions.
>> > > > > > > >
>> > > > > > > > Regards,
>> > > > > > > > Saikat
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Fri, Oct 11, 2019 at 4:56 PM Emmanouil Gkatziouras
<
>> > > > > > > > gkatzioura@gmail.com>
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Hi all,
>> > > > > > > > >
>> > > > > > > > > By mentioning Pub/Sub I mean Google Cloud's
Pub/Sub [1]
>> > > > > > > > > Yes it is indeed cloud specific. It is the
messaging
>> service
>> > > > > provided
>> > > > > > > by
>> > > > > > > > > GCP.
>> > > > > > > > > I have made the integration with it, but
I haven't
>> published
>> > it
>> > > > yet
>> > > > > > (I
>> > > > > > > am
>> > > > > > > > > waiting for the discussion/ticket and proceed
with the
>> > > > > contribution).
>> > > > > > > > >
>> > > > > > > > > Kind regards,
>> > > > > > > > > Emmanouil
>> > > > > > > > >
>> > > > > > > > > [1] https://cloud.google.com/pubsub/docs/overview
>> > > > > > > > >
>> > > > > > > > > *Emmanouil Gkatziouras*
>> > > > > > > > > https://egkatzioura.com/ |
>> > > > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
>> > > > > > > > > https://github.com/gkatzioura
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Fri, 11 Oct 2019 at 21:45, Denis Magda
<
>> dmagda@apache.org
>> > >
>> > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > Hello Emmanouil,
>> > > > > > > > > >
>> > > > > > > > > > Are you referring to any specific implementation
>> tailored
>> > > for a
>> > > > > > > cloud?
>> > > > > > > > If
>> > > > > > > > > > I'm not mistaken you were mentioning
AWS/GCE. Is there
>> any
>> > > > Github
>> > > > > > > > source
>> > > > > > > > > > code?
>> > > > > > > > > >
>> > > > > > > > > > Here is a list of existing integrations
with streaming
>> and
>> > > > > pub/sub
>> > > > > > > > > > frameworks:
>> > > > > > > > > > https://apacheignite-mix.readme.io/docs/overview
>> > > > > > > > > >
>> > > > > > > > > > with the source code in Ignite master:
>> > > > > > > > > > https://github.com/apache/ignite/tree/master/modules
>> > > > > > > > > >
>> > > > > > > > > > Check out those resources to see what
are the most
>> suitable
>> > > > > Ignite
>> > > > > > > APIs
>> > > > > > > > > for
>> > > > > > > > > > such integrations.
>> > > > > > > > > >
>> > > > > > > > > > @Saikat Maitra <saikat.maitra@gmail.com>
, @Roman <
>> > > > > > rshtykh@yahoo.com
>> > > > > > > >
>> > > > > > > > ,
>> > > > > > > > > as
>> > > > > > > > > > streaming technologies contributors/maintainers,
could
>> you
>> > > > please
>> > > > > > > join
>> > > > > > > > > this
>> > > > > > > > > > thread?
>> > > > > > > > > >
>> > > > > > > > > > -
>> > > > > > > > > > Denis
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > On Thu, Oct 10, 2019 at 7:58 AM Emmanouil
Gkatziouras <
>> > > > > > > > > > gkatzioura@gmail.com>
>> > > > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > Hello everyone,
>> > > > > > > > > > >
>> > > > > > > > > > > I started using Ignite lately.
Part of my work
>> involves
>> > > > > Pub/Sub,
>> > > > > > > > thus I
>> > > > > > > > > > > created a pub/sub streamer.
>> > > > > > > > > > > Pub/Sub is pretty close to Kafka.
>> > > > > > > > > > > Here's how the streamer works.
Every node creates an
>> > > instance
>> > > > > of
>> > > > > > a
>> > > > > > > > > > Pub/Sub
>> > > > > > > > > > > client.
>> > > > > > > > > > > The clients subscribe to the same
topic using the same
>> > > > > > subscription
>> > > > > > > > > name.
>> > > > > > > > > > > A message is being sent and one
of the nodes will
>> receive
>> > > the
>> > > > > > > > message,
>> > > > > > > > > > > process it and then forward it
to Ignite using the
>> > > extractor
>> > > > > > > > provided.
>> > > > > > > > > > >
>> > > > > > > > > > > Any thoughts?
>> > > > > > > > > > >
>> > > > > > > > > > > *Emmanouil Gkatziouras*
>> > > > > > > > > > > https://egkatzioura.com/ |
>> > > > > > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/
>> > > > > > > > > > > https://github.com/gkatzioura
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message