flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Timo Walther <twal...@apache.org>
Subject Re: Optimizing multiple aggregate queries on a CEP using Flink
Date Mon, 12 Feb 2018 16:31:48 GMT
Hi Sahil,

I'm not a CEP expert but I will loop in Kostas (in CC). In general, the 
example that you described can be easily done with a ProcessFunction 
[1]. A process function not only allows to keep state (like a count) but 
also allows you to set timers flexibly for specific use cases such that 
aggregations can be triggered/reused. So in general I would say that 
implementing and testing such an algorithm is possible. How easy it can 
be interegrated into the CEP API, I don't know.

Regards,
Timo



[1] 
https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/operators/process_function.html

Am 2/9/18 um 11:28 PM schrieb Sahil Arora:
> Hi there,
> We have been working on a project with the title "Optimizing Multiple 
> Aggregate Queries over a Complex Event Processing Engine". The aim is 
> to optimize a group of queries. Take such as*"how many cars passed the 
> post in the past 1 minute" *and*"how many cars passed the post in the 
> past 2 minutes"* are 2 queries, and the naive and inefficient method 
> to answer both the queries is to independently solve both of these 
> queries one by one and find the answer. However, the optimum way would 
> be to minimize the computation by using the answer given by query 1 
> and using it in query 2. This is basically what our aim is, to 
> minimize computation cost when we have multiple aggregate queries in a 
> CEP.
>
> We have been searching for some platform which supports CEP, and Flink 
> is probably one of them. Hence, it would be very helpful if we could 
> get some answers to the following questions:
>
> 1. Does flink already have some method of optimizing multiple 
> aggregate queries?
> 2. Is it possible for us to implement / test such an algorithm in 
> flink which considers multiple queries in a CEP, like having a 
> database of SQL queries and testing an algorithm of our choice?
>
> Any other inputs which may help us with solving the problem would be 
> highly welcome.
>
> Thanks a lot.
> -- 
> Sahil Arora
> Final year B.Tech Undergrad | Indian Institute of Technology Mandi
> Web: https://sahilarora535.github.io
> LinkedIn: sahilarora535 <https://www.linkedin.com/in/sahilarora535/>
> Ph: +91-8130506047 <tel:+91%2081305%2006047>



Mime
View raw message