flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: Process event with last 1 hour, 1week and 1 Month data
Date Thu, 15 Jun 2017 09:41:03 GMT

How would you evaluate such a query? I think the answer could be that you have to keep all
that older data around so that you can evaluate when a new event arrives. In Flink, you could
use a ProcessFunction for that and use a MapState that keeps events bucketed into one-week
intervals. This way, can more efficiently iterate over the buckets that are required when
evaluating a given event and you can also efficiently delete a complete bucket of older events
once you know that they are not required anymore.

These are the relevant sections of the Flink doc:
 - https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/stream/state.html <https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/stream/state.html>
 - https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/stream/process_function.html

> On 13. Jun 2017, at 15:27, shashank agarwal <shashank734@gmail.com> wrote:
> Hi,
> I have to process each event with last 1 hour , 1 week and 1 month data. Like how many
times same ip occurred in last 1 month corresponding to that event. \
> I think window is for fixed time i can't calculate with last 1 hour corresponding to
current event.
> If you have any clue please guide what should i use Table, ProcessFunction or global
window. Or what approach should i take ?
> -- 
> Thanks Regards
>  ---  Trying to mobilize the things....

View raw message