Mailing-List: contact dev-help@gearpump.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@gearpump.incubator.apache.org
MIME-Version: 1.0
References: <CAH3zwfL=k_3VEF=w_DDULEKc56LS9+Pozwrtgu5VxCsEWfyqOA@mail.gmail.com>
 <TY1PR0301MB11515FCB4160E7BA12B7EEDBA87C0@TY1PR0301MB1151.apcprd03.prod.outlook.com>
In-Reply-To: <TY1PR0301MB11515FCB4160E7BA12B7EEDBA87C0@TY1PR0301MB1151.apcprd03.prod.outlook.com>
From: Manu Zhang <owenzhang1990@gmail.com>
Date: Thu, 05 May 2016 04:38:33 +0000
Message-ID: <CABT57mYiuE4Tp3iA_7C44_U_3V_9bZ6=VUSZk-tagN7gkf0BYQ@mail.gmail.com>
Subject: Re: Sink/source tickets
To: "dev@gearpump.incubator.apache.org" <dev@gearpump.incubator.apache.org>
Content-Type: multipart/alternative; boundary=001a113cf4f0c0c5bd053210ea90
archived-at: Thu, 05 May 2016 04:38:48 -0000

--001a113cf4f0c0c5bd053210ea90
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

+1

I need to do some PoC for GEARPUMP-116. It's not difficult to wrap a Storm
spout/bolt in Gearpump task since we have very similar flow for task
executor. The hard thing is about message delivery guarantee and state
management where the two diverge. Also, we have no control over cpu/memory
usage and thus performance.

Thanks,
Manu

On Thu, May 5, 2016 at 11:51 AM Jiang Weihua <whjiang@outlook.com> wrote:

> +1 on the order.
>
> @manuzhang, how difficult to have GEARPUMP-116 as you wrote the Storm
> compatibility layer?
>
> NOTE: GEARPUMP-116 is DIFFERENT from our Storm app compatibility.
> GEARPUMP-116 tries to provide Storm compatible Source/Sink to Gearpump
> native application instead of Storm app.
>
> Thanks
> Weihua
>
>
>
> =E5=9C=A8 16/5/5 =E4=B8=8A=E5=8D=881:34=EF=BC=8C=E2=80=9CKarol Brejna=E2=
=80=9D<karol.brejna@gmail.com> =E5=86=99=E5=85=A5:
>
> >We have a series of jira tickets regarding Gearpump sinks/sources:
> >
> >https://issues.apache.org/jira/browse/GEARPUMP-116 - Compatibility
> >layer/adapter for Apache Storm
> >https://issues.apache.org/jira/browse/GEARPUMP-115 - Create MQTT
> source/sink
> >https://issues.apache.org/jira/browse/GEARPUMP-106 - Gearpump Redis
> >Integration
> >https://issues.apache.org/jira/browse/GEARPUMP-105 - Provide
> non-persistent
> >Sink Task so that examples like word count can materialize Sum results
> >within the Client
> >https://issues.apache.org/jira/browse/GEARPUMP-100 - Source task that
> emits
> >messages per a schedule (interval or otherwise) should be provided
> >https://issues.apache.org/jira/browse/GEARPUMP-95 - Add parquet
> datasource
> >and datasink connectors
> >https://issues.apache.org/jira/browse/GEARPUMP-91 - Apache Cassandra
> >Integration
> >
> >We also had a ticket for 'Add a HDFS Sink with secutiry' (
> >https://github.com/gearpump/gearpump/issues/1547) - I am not sure as for
> >the outcome of this one.
> >
> >Most of them consider the medium (MQTT, Redis, Casandra, ...). Other tal=
k
> >about the source mechanics (scheduled/repetative source).
> >
> >I'd like to discuss the order in wich we plan implementation for them.
> >
> >In my opinion Redis an MQTT (GEARPUMP-106, GEARPUMP-115) seems most
> >important to have.
> >Redis is well known and widely used. MQTT is a de facto standard in IoT
> >communications.
> >
> >Then I would like to have HDFS sink (if we didn't merged this already).
> >
> >Non-persistent datasink could be very useful for examples/demo purposes.
> >(Imagine we have capped collection that the application can send message=
s
> >to, kind of application console. In the dashboard there could be a secti=
on
> >that presents lates 'console' messages. This way a user could "watch" th=
e
> >application progress. Especially if he/she doesn't have access to the
> >backend - as it happens often in YARN mode. But this is a topic for
> >dedicated discussion, I think.)
> >
> >On the other hand, if we start working on GEARPUMP-116, we'd probably
> >quickly have Redis, JMS, AMQP sources (adapted from Storm)
> >
> >Please, let me know what do you think.
> >
> >Karol
>
>

--001a113cf4f0c0c5bd053210ea90--