From dev-return-48029-archive-asf-public=cust-asf.ponee.io@ignite.apache.org Sat Oct 19 02:03:44 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 2880B1804BB for ; Sat, 19 Oct 2019 04:03:44 +0200 (CEST) Received: (qmail 29440 invoked by uid 500); 19 Oct 2019 02:03:42 -0000 Mailing-List: contact dev-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ignite.apache.org Delivered-To: mailing list dev@ignite.apache.org Received: (qmail 29145 invoked by uid 99); 19 Oct 2019 02:03:41 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 19 Oct 2019 02:03:41 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id B78C5C1CFA for ; Sat, 19 Oct 2019 02:03:40 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.002 X-Spam-Level: *** X-Spam-Status: No, score=3.002 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_REPLY=1, HTML_MESSAGE=0.2, KAM_BADIPHTTP=2, NORMAL_HTTP_TO_IP=0.001, NUMERIC_HTTP_ADDR=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-ec2-va.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id 1AjTm49oDEvq for ; Sat, 19 Oct 2019 02:03:38 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.208.195; helo=mail-lj1-f195.google.com; envelope-from=saikat.maitra@gmail.com; receiver= Received: from mail-lj1-f195.google.com (mail-lj1-f195.google.com [209.85.208.195]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id 15771BC5B7 for ; Sat, 19 Oct 2019 02:03:38 +0000 (UTC) Received: by mail-lj1-f195.google.com with SMTP id a22so8004501ljd.0 for ; Fri, 18 Oct 2019 19:03:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=duYcnOQY/kT66VZgVXlvCynYIFJYmMD32kqyutt9X7E=; b=TSLzGHK4eUvzoFOzh2cuUmDlLOSt2ck913z4PEzOVSHs1qiWKPgo5f+ME4TvCqqB9U EVOuEiGWyr+hk3bwy/HG8FWpZA6n1TgSINht4jK7KKWdVY6D9YvGh50CMI8mVBWKpcHT xqv4qpUQEMxaahCEhop6W6IFUCgrP8VtcMve9JNQyfnm9SLW2w+dRjOZS6x9Rw2vAEqC PXK0X915YY8R1pgUINPKY9tPON+Xr6T+w3/IEi5ohw7nb5rGIKy/fWHNOlfwlMnFr2aT S4OBkeDumxZYwpAeE/CYahcXKkPGV/sCLEd6TPkXK31rWCtQ/5qgO0bRNWZ494mJSTNr 73mQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=duYcnOQY/kT66VZgVXlvCynYIFJYmMD32kqyutt9X7E=; b=gAPuE9C9jCQa03TpVVYcVvWD52B5grgEnVzvXBgshj8z0um+xU54zONy5MRDudATf5 ugXXi8Zx89/BTUZBMBWCQS1YcCQ+DPQ/3Z2vD4WPTdCVnBEucFIWQWvF4asVS50M37UZ pOj2TX7w+kcgfEOEjpH8KziRphsf75VGje3kYRvZOLh9MFuCm4jzHmwVw6Q27TD5RqEr AbaPJ3YB9l202Q6BDCso/nJjEejzSaCRxEaqSCJGMuvJOI7TayNu/gNgzQCvSgAiO/rC TrQgotviag2MpWry7mjktfY/824Oh9eM/UcFqfujFMdp5RpfDYrdv3zn0t5RxlIbmvk5 mBGw== X-Gm-Message-State: APjAAAUsImEyebs0S6GrKabcaqi6W89d1s3YRO3l0v7g5TEPq/8HU8HZ 5jlRIKbjxtJ24B2ILXk5eGHuJPNYa4zU0qHO/dRCT101fNQ= X-Google-Smtp-Source: APXvYqz9eCwHG3fNksdr98ry0IMDkOzFYz2NbDrikVm3c3PSxrIE/CTsD2Ubqp0GEhK1os5nR0EwPeObMuXiifrJM3U= X-Received: by 2002:a2e:7e05:: with SMTP id z5mr7971105ljc.224.1571450610296; Fri, 18 Oct 2019 19:03:30 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Saikat Maitra Date: Fri, 18 Oct 2019 21:03:18 -0500 Message-ID: Subject: Re: [DISCUSS] Pub/Sub Streamer Implementation To: dev@ignite.apache.org Content-Type: multipart/alternative; boundary="00000000000068a9f2059539dbaa" --00000000000068a9f2059539dbaa Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello Denis, Sure, sounds good. I will create a separate discussion thread to get community feedback on whether we should join the Bahir or kick-off "Ignite Extensions" project. Regards, Saikat On Thu, Oct 17, 2019 at 2:21 PM Denis Magda wrote: > Folks, > > The concept of Apache Bahir is precisely what we need! Saikat, thanks for > doing the research. Why I believe the * idea* fits us well: > > - All integrations can be stored in separate Github repositories and > have their dev lifecycles. We've not obliged to couple all the > integrations > in a single repo. For instance, Spark can be located in a dedicated re= po > while streaming integrations might be in a single one. It's up to us. > - All the repositories and integrations belong to ASF. We're not dumpi= ng > them on Github but rather keep supporting and developing in accordance > with > ASF vision and practices. > - There is a way to reward contributors via committership and PMC > membership. You won't get this if a project is just one of the million= s > of > Github projects. > > Saikat, as Ignite PMC member, could you please kick-off a separate dev-li= st > discussion to involve more community members and referring to this thread= ? > I think the primary question the community needs to answer whether we > should join the Bahir or kick-off "Ignite Extensions" project? > > - > Denis > > > On Thu, Oct 17, 2019 at 2:54 AM Alexey Zinoviev > wrote: > > > Maybe we could move all our Streaming Integrations there, but what is > about > > maintaining and committer permissions to the new repositories? > > I see the list of the committers and PMC members there > > https://bahir.apache.org/community-members/ > > > > Could somebody from Ignite community be added to this list as a > > PMC/committer to maintain Ignite integrations? > > If yes, I'd happy to join this project and collaborate with these guys > > together > > If no, and we should start from the zero level with external PRs - hmmm= , > > it's better to have our own external repository with ApacheBahirr > ideology. > > > > Also, I agree, that all connectors could be moved there (in ApacheBahir= r > > like repository), but not all modules from the page > > > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularizati= on#IEP-36:Modularization-IndependentIntegrations > > because > > they use different parts of core modules and could be developed with > > different velocity. > > > > In reality, before creation of new repositories we need wide discussion > > based on the document mentioned above. > > > > P.S. Of course, we could experiment in the next release 2.8 with one or > two > > integrations like twitter or ZeroMQ > > Also, I'd like Denis Magda idea for the separate repositories in Apache > > Github to maintain them by anybody. > > > > > > =D1=87=D1=82, 17 =D0=BE=D0=BA=D1=82. 2019 =D0=B3. =D0=B2 05:02, Saikat = Maitra : > > > > > Hi Denis, Emmanouil > > > > > > Thank you for your email. I think creating a separate integration > project > > > in github and also proposing it as an apache incubator project will b= e > a > > > good move. The other separate integration modules > > > > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularizati= on#IEP-36:Modularization-IndependentIntegrations > > > can > > > be moved to this new apache incubator project. > > > > > > There are similar integration available for Flink and Spark in Apache > > Bahir > > > https://bahir.apache.org/ > > > > > > Do you think we should reach out to Apache Bahir community with a > > proposal > > > or we should plan to create a separate Apache Incubator project? > > > > > > Regards, > > > Saikat > > > > > > > > > > > > On Wed, Oct 16, 2019 at 6:21 AM Emmanouil Gkatziouras < > > > gkatzioura@gmail.com> > > > wrote: > > > > > > > Hi all! > > > > > > > > Based on the discussions most probably this is going to be on anoth= er > > > > GitHub repo. Therefore I will prepare a standalone project with the > > > Pub/Sub > > > > module and once the repository is created a pull request shall be > > issued. > > > > If there is anything else I can do in order to facilitate this plea= se > > let > > > > me know. > > > > > > > > If a ticket is created for this issue, I shall be able to create a > pull > > > > request and thus a build will take place on ignite's CI. > > > > If everyone is aligned with adding a Pub/Sub implementation may I > > create > > > a > > > > ticket on JIRA? > > > > > > > > My next question has to do on how Ignite configures the build jobs = in > > > Team > > > > City. The unit tests for my implementation need a local Pub/Sub > server > > up > > > > and running (they provide one for testing purposes). Will this take > > > effect > > > > in a form of a build script inside my implementation or is it > something > > > > that should be configured on Team City? > > > > > > > > Kind regards, > > > > Emmanouil > > > > > > > > > > > > *Emmanouil Gkatziouras* > > > > https://egkatzioura.com/ | > > > > https://www.linkedin.com/in/gkatziourasemmanouil/ > > > > https://github.com/gkatzioura > > > > > > > > > > > > On Tue, 15 Oct 2019 at 23:27, Denis Magda wrote= : > > > > > > > > > Emmanouil, Saikat, > > > > > > > > > > After contemplating on this, let me propose another way with this > > > > > particular integration that is aligned with our modularization > > > endeavor. > > > > > The modularization [1] defines the Ignite core, Ignite modules, a= nd > > > > > independent integrations. The first two are to be located in Igni= te > > > > > repositories and to be tested/updated by the community for every > > > release. > > > > > The modules are a subset of existing integrations that are adopte= d > > > widely > > > > > and require community attention. > > > > > > > > > > As for the independent integrations [1], those are supposed to be > > > located > > > > > in a separate GitHub repo. I would advise us to start implementin= g > > this > > > > > concept with the Pub/Sub integration. Even though it will be > located > > > in a > > > > > separate Github repo, it will be listed among officially availabl= e > > > > > integrations on the Ignite website featuring Emmanouil as a > > > contributor. > > > > > > > > > > What do you think? > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularizati= on#IEP-36:Modularization-IndependentIntegrations > > > > > > > > > > > > > > > > > > > > - > > > > > Denis > > > > > > > > > > > > > > > On Mon, Oct 14, 2019 at 9:47 PM Saikat Maitra < > > saikat.maitra@gmail.com > > > > > > > > > wrote: > > > > > > > > > > > Hi Emmanouil, > > > > > > > > > > > > The changes looks good to me. I wanted to check if you are also > > able > > > to > > > > > > validate that the once data is added to Ignite cluster you are > also > > > > able > > > > > to > > > > > > access it using another Ignite client or using rest endpoints. > > > > > > > > > > > > We use Teamcity for CI process, the details are as follows > > > > > > > > > > > > Trigger validation of those test suites that have been affected > by > > > your > > > > > > changes on TeamCity : < > > > > > http://204.14.53.153/ > > > > > > > > > > > > > > > > > > > - Locate a test suite you have to check, press button named > > "..." > > > > that > > > > > > is located on the left of "Run" button. "Run custom build" > > dialog > > > > will > > > > > > appear; > > > > > > - Go to "Changes" tab and choose > > "pull//head" > > > > in > > > > > > "Build branch" dropdown list; > > > > > > - Press "Run build" button and monitor tests results. > > > > > > > > > > > > > > > > > > > > > > > > @Roman rshtykh@yahoo.com - can you also please take a look? > > > > > > > > > > > > Regards, > > > > > > Saikat > > > > > > > > > > > > On Sun, Oct 13, 2019 at 6:20 PM Emmanouil Gkatziouras < > > > > > > gkatzioura@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > Hi @Saikat Maitra > > > > > > > > > > > > > > I have my implementation on an Ignite fork on my personal > GitHub > > > > > account > > > > > > > [1] > > > > > > > It is based on the Kafka implementation which is of a polling > > > nature. > > > > > > > GCP does not provide a service like Kafka. Instead it provide= s > us > > > > with > > > > > > > Pub/Sub which is just like Kafka. > > > > > > > Therefore using the Kafka streamer as a guideline made more > sense > > > to > > > > > me. > > > > > > > > > > > > > > Apart from the implementation I have also added a unit test. > > > > > > > It definitely needs some polishing and a double check on my > side > > to > > > > > make > > > > > > > the review smooth. > > > > > > > For the tests a local Pub/Sub [2] server has to be setup > however > > I > > > am > > > > > not > > > > > > > sure which is the best way to do so with regards to your CI/C= D. > > > > > > > > > > > > > > [1] > > > https://github.com/gkatzioura/ignite/tree/pubsub/modules/pubsub > > > > > > > [2] https://egkatzioura.com/2019/09/22/pub-sub-local-emulator= / > > > > > > > > > > > > > > *Emmanouil Gkatziouras* > > > > > > > https://egkatzioura.com/ | > > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/ > > > > > > > https://github.com/gkatzioura > > > > > > > > > > > > > > > > > > > > > On Sat, 12 Oct 2019 at 02:11, Saikat Maitra < > > > saikat.maitra@gmail.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Hello Emmanouil, > > > > > > > > > > > > > > > > Can you please share more info about the implementation? > > > > > > > > > > > > > > > > We have similar implementation for data streamers with Kafk= a > > and > > > > > > RocketMQ > > > > > > > > > > > > > > > > https://apacheignite-mix.readme.io/docs/kafka-streamer > > > > > > > > > > > > > > > > https://apacheignite-mix.readme.io/docs/rocketmq-streamer > > > > > > > > > > > > > > > > Please review and share if you have any questions. > > > > > > > > > > > > > > > > Regards, > > > > > > > > Saikat > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Oct 11, 2019 at 4:56 PM Emmanouil Gkatziouras < > > > > > > > > gkatzioura@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > By mentioning Pub/Sub I mean Google Cloud's Pub/Sub [1] > > > > > > > > > Yes it is indeed cloud specific. It is the messaging > service > > > > > provided > > > > > > > by > > > > > > > > > GCP. > > > > > > > > > I have made the integration with it, but I haven't > published > > it > > > > yet > > > > > > (I > > > > > > > am > > > > > > > > > waiting for the discussion/ticket and proceed with the > > > > > contribution). > > > > > > > > > > > > > > > > > > Kind regards, > > > > > > > > > Emmanouil > > > > > > > > > > > > > > > > > > [1] https://cloud.google.com/pubsub/docs/overview > > > > > > > > > > > > > > > > > > *Emmanouil Gkatziouras* > > > > > > > > > https://egkatzioura.com/ | > > > > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/ > > > > > > > > > https://github.com/gkatzioura > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, 11 Oct 2019 at 21:45, Denis Magda < > dmagda@apache.org > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hello Emmanouil, > > > > > > > > > > > > > > > > > > > > Are you referring to any specific implementation tailor= ed > > > for a > > > > > > > cloud? > > > > > > > > If > > > > > > > > > > I'm not mistaken you were mentioning AWS/GCE. Is there > any > > > > Github > > > > > > > > source > > > > > > > > > > code? > > > > > > > > > > > > > > > > > > > > Here is a list of existing integrations with streaming > and > > > > > pub/sub > > > > > > > > > > frameworks: > > > > > > > > > > https://apacheignite-mix.readme.io/docs/overview > > > > > > > > > > > > > > > > > > > > with the source code in Ignite master: > > > > > > > > > > https://github.com/apache/ignite/tree/master/modules > > > > > > > > > > > > > > > > > > > > Check out those resources to see what are the most > suitable > > > > > Ignite > > > > > > > APIs > > > > > > > > > for > > > > > > > > > > such integrations. > > > > > > > > > > > > > > > > > > > > @Saikat Maitra , @Roman < > > > > > > rshtykh@yahoo.com > > > > > > > > > > > > > > > > , > > > > > > > > > as > > > > > > > > > > streaming technologies contributors/maintainers, could > you > > > > please > > > > > > > join > > > > > > > > > this > > > > > > > > > > thread? > > > > > > > > > > > > > > > > > > > > - > > > > > > > > > > Denis > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Oct 10, 2019 at 7:58 AM Emmanouil Gkatziouras < > > > > > > > > > > gkatzioura@gmail.com> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hello everyone, > > > > > > > > > > > > > > > > > > > > > > I started using Ignite lately. Part of my work involv= es > > > > > Pub/Sub, > > > > > > > > thus I > > > > > > > > > > > created a pub/sub streamer. > > > > > > > > > > > Pub/Sub is pretty close to Kafka. > > > > > > > > > > > Here's how the streamer works. Every node creates an > > > instance > > > > > of > > > > > > a > > > > > > > > > > Pub/Sub > > > > > > > > > > > client. > > > > > > > > > > > The clients subscribe to the same topic using the sam= e > > > > > > subscription > > > > > > > > > name. > > > > > > > > > > > A message is being sent and one of the nodes will > receive > > > the > > > > > > > > message, > > > > > > > > > > > process it and then forward it to Ignite using the > > > extractor > > > > > > > > provided. > > > > > > > > > > > > > > > > > > > > > > Any thoughts? > > > > > > > > > > > > > > > > > > > > > > *Emmanouil Gkatziouras* > > > > > > > > > > > https://egkatzioura.com/ | > > > > > > > > > > > https://www.linkedin.com/in/gkatziourasemmanouil/ > > > > > > > > > > > https://github.com/gkatzioura > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --00000000000068a9f2059539dbaa--