flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hao Chen <...@apache.org>
Subject Re: Apache siddhi into Flink
Date Mon, 29 Aug 2016 09:26:19 GMT
+1 support siddhi as a flink operator on DataStream, siddhi support rich
CEP features other than pattern match, also support extensible
snapshot/restore interface for fault-tolerance which should be easy to
integrate with flink's state management (tested with some draft code), we
(apache eagle community) use siddhi a lot to integrate with Storm and Spark
for real-time monitoring cases. And siddhi's community is active as well in
releasing, bug fixing, and accepting external contribution etc.

- Hao

On Mon, Aug 29, 2016 at 5:21 PM, Till Rohrmann <trohrmann@apache.org> wrote:

> Hi Aparup,
>
> I haven't looked in detail at Siddhi's internals especially the way it
> handles distributed execution. I've only seen that it uses Hazelcast for a
> distributed in-memory cache. If a distributed cache for the communication
> between instances of the CEP operator is needed, then you would have to
> integrate that as well. Otherwise, if the state stored in the operators is
> disjunct, then you should be good to go with Flink's state abstraction.
>
> If you want to work on the integration, then it's best to open a JIRA
> issue and describe a little bit the way the integration could look like.
>
> Cheers,
> Till
>
> On Mon, Aug 29, 2016 at 10:59 AM, Stephan Ewen <sewen@apache.org> wrote:
>
>> Nice idea!
>>
>> If you look at the current CEP library, it is simply a custom operator.
>> Often, you can even get away with a custom FlatMapFunction that uses
>> state: https://ci.apache.org/projects/flink/flink-docs-master/dev/
>> state.html#using-the-keyvalue-state-interface
>>
>> Stephan
>>
>>
>> On Mon, Aug 29, 2016 at 10:09 AM, Aparup Banerjee (apbanerj) <
>> apbanerj@cisco.com> wrote:
>>
>>> I think siddhi is a fairly matured CEP library. I am thinking it should
>>> co-exist with existing CEP library. My thinking is we should be able to use
>>> Siddhi QL/ Siddhi Patterns on top of flink data streams. This can co-exist
>>> naturally with existing Java / Scala based Flink CEP Library. I am still
>>> reading up on Flink internals – but at high level I am thinking about a new
>>> Flink operator on DataStream for this. Thoughts ?
>>>
>>> Thanks,
>>> Aparup
>>>
>>> From: Chesnay Schepler <chesnay@apache.org>
>>> Reply-To: "user@flink.apache.org" <user@flink.apache.org>
>>> Date: Monday, August 29, 2016 at 12:35 AM
>>> To: "user@flink.apache.org" <user@flink.apache.org>
>>> Subject: Re: Apache siddhi into Flink
>>>
>>> Hello Aparup,
>>>
>>> could you provide more information about Siddhi? How mature is it; how
>>> is the community? How does it compare to the Flink's CEP library?
>>>
>>> How should this integration look like? Are you proposing to replace the
>>> current CEP library, or will they co-exist with different use-cases for
>>> each?
>>>
>>> If we used Siddhi in Flink, how exactly would Flink's runtime be
>>> involved in the processing?
>>>
>>> Regards,
>>> Chesnay
>>>
>>> On 28.08.2016 23:21, Aparup Banerjee (apbanerj) wrote:
>>>
>>> Sorry for the semantic difference.
>>>
>>>
>>>
>>> On Aug 28, 2016, at 12:05 PM, Trevor Grant < <trevor.d.grant@gmail.com>
>>> trevor.d.grant@gmail.com> wrote:
>>>
>>> Thank you for confirming Hao,
>>>
>>> Aparup, please don't refer to it as "Apache Siddhi", that is
>>> misleading.
>>>
>>>
>>> Trevor Grant
>>> Data Scientist
>>> https://github.com/rawkintrevo
>>> http://stackexchange.com/users/3002022/rawkintrevo
>>> http://trevorgrant.org
>>>
>>> *"Fortunate is he, who is able to know the causes of things."  -Virgil*
>>>
>>>
>>> On Sun, Aug 28, 2016 at 10:50 AM, Hao Chen <hao@apache.org> wrote:
>>>
>>>> Siddhi is not apache project, but licensed under apache license v2,
>>>> being open sourced and maintained by wso2.
>>>>
>>>> - Hao
>>>>
>>>> On Sun, Aug 28, 2016 at 11:11 PM, Trevor Grant <
>>>> trevor.d.grant@gmail.com> wrote:
>>>>
>>>>> Aparup,
>>>>>
>>>>> Was Siddhi recently added as an incubator project?  I can't find it in
>>>>> the project directory or or on github.com/apache.  The closest thing
>>>>> I can find is this:  <https://github.com/wso2/siddhi>
>>>>> https://github.com/wso2/siddhi
>>>>>
>>>>> tg
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Trevor Grant
>>>>> Data Scientist
>>>>> <https://github.com/rawkintrevo>https://github.com/rawkintrevo
>>>>> <http://stackexchange.com/users/3002022/rawkintrevo>
>>>>> http://stackexchange.com/users/3002022/rawkintrevo
>>>>> <http://trevorgrant.org>http://trevorgrant.org
>>>>>
>>>>> *"Fortunate is he, who is able to know the causes of things."  -Virgil*
>>>>>
>>>>>
>>>>> On Sat, Aug 27, 2016 at 5:36 PM, Chen Qin <qinnchen@gmail.com>
wrote:
>>>>>
>>>>>> ​+1​
>>>>>>
>>>>>>
>>>>>> On Aug 26, 2016, at 11:23 PM, Aparup Banerjee (apbanerj) <
>>>>>> <apbanerj@cisco.com>apbanerj@cisco.com> wrote:
>>>>>>
>>>>>> Hi-
>>>>>>
>>>>>> Has anyone looked into embedding apache siddhi into Flink.
>>>>>>
>>>>>> Thanks,
>>>>>> Aparup
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>
>

Mime
View raw message