flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abdul Salam Shaikh <abd.salam.sha...@gmail.com>
Subject Re: Regarding time window based on the values received in the stream
Date Fri, 18 Nov 2016 14:59:08 GMT
Hello Mr Hueske,

Thank you for reaching out to my query.

The example stated in the documentation is the same use case for me where I
am trying to build a prototype regarding a traffic metric in Germany as a
part of my thesis.

The data is received from multiple detectors and there is a field which
measures the current cycle time(Umlaufsekunde) in the whole cycle.  So, as
far as my analysis I haven't seen any event as of now where the previous
Umlaufsekunde greater than the new one.

So if I understand you correctly,

 .keyBy(<key selector>)
    .window(<window assigner>)  --- Global Window
    .trigger(<trigger>)         --- Indicates when the collected data
is ready to be processed
    .<windowed transformation>(<window function>); --- Actual data processing

Could you point me to some repo(if you know of any) which has examples of
implementation of Triggers and Windows ?

Also, will the concept of generating timestamps/watermarks be applicable in
this scenario ?

On Fri, Nov 18, 2016 at 9:50 AM, Fabian Hueske <fhueske@gmail.com> wrote:

> Hi,
> that does not sound like a time window problem because there is not
> time-related condition to split the windows.
> I think you can implement that with a GlobalWindow and a custom trigger.
> The documentation about global windows, triggers, and evictors [1] and this
> blogpost [2] might be helpful
> On thing that came to my mind reading your description, are you sure that
> the events arrive in the correct order?
> Best,
> Fabian
> [1] https://ci.apache.org/projects/flink/flink-docs-
> release-1.1/apis/streaming/windows.html
> [2] http://flink.apache.org/news/2015/12/04/Introducing-windows.html
> 2016-11-18 1:04 GMT+01:00 Abdul Salam Shaikh <abd.salam.shaikh@gmail.com>:
>> Hi,
>> I have a requirement for my thesis project where I need to set the time
>> window based on the value which I received in the event stream.
>> I need to collect the values when the value starts at 0 and increments to
>> a max value(which is unknown) till the value resets to 0 again on which a
>> trigger is triggered which performs a calculation based on the values
>> collected.
>> DataStream<Tuple4<String,Double,String,Integer>> detectorEventStream
>> streamMatchedTuples
>>         .keyBy(Based on the first param of the tuple)
>>         .timeWindow(based on the second param of the tuple)
>>         .apply(new PhaseCalculator());
>> I am relatively new in flink so I would like to have ideas on how to
>> proceed.
>> --
>> Thanks & Regards,
>> *Abdul Salam Shaikh*

Thanks & Regards,

*Abdul Salam Shaikh*

View raw message