flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@gmail.com>
Subject Re: Regarding time window based on the values received in the stream
Date Mon, 28 Nov 2016 10:17:27 GMT
Hi,

sorry for the late reply.

There is a repository [1] with an example application that uses a custom
trigger [2] (though together with a TimeWIndow and not with a GlobalWindow).
I'm not aware of a repo with an example of a GlobalWIndow.

Regarding the question about timestamps and watermarks: It depends if you
need them or not. If you can be sure that your data always arrives in
order, you might be good without using timestamps and watermarks.
Otherwise, it might make sense to set them, but then you would need to sort
the data on time before you pass it into the global window.

Best,
Fabian

[1] https://github.com/dataArtisans/flink-streaming-demo
[2]
https://github.com/dataArtisans/flink-streaming-demo/blob/master/src/main/scala/com/dataartisans/flink_demo/examples/EarlyArrivalCount.scala

2016-11-18 15:59 GMT+01:00 Abdul Salam Shaikh <abd.salam.shaikh@gmail.com>:

> Hello Mr Hueske,
>
> Thank you for reaching out to my query.
>
> The example stated in the documentation is the same use case for me where
> I am trying to build a prototype regarding a traffic metric in Germany as a
> part of my thesis.
>
> The data is received from multiple detectors and there is a field which
> measures the current cycle time(Umlaufsekunde) in the whole cycle.  So, as
> far as my analysis I haven't seen any event as of now where the previous
> Umlaufsekunde greater than the new one.
>
> So if I understand you correctly,
>
>  .keyBy(<key selector>)
>     .window(<window assigner>)  --- Global Window
>     .trigger(<trigger>)         --- Indicates when the collected data is ready
to be processed
>     .<windowed transformation>(<window function>); --- Actual data processing
>
> Could you point me to some repo(if you know of any) which has examples of
> implementation of Triggers and Windows ?
>
> Also, will the concept of generating timestamps/watermarks be applicable
> in this scenario ?
>
>
> On Fri, Nov 18, 2016 at 9:50 AM, Fabian Hueske <fhueske@gmail.com> wrote:
>
>> Hi,
>>
>> that does not sound like a time window problem because there is not
>> time-related condition to split the windows.
>> I think you can implement that with a GlobalWindow and a custom trigger.
>> The documentation about global windows, triggers, and evictors [1] and this
>> blogpost [2] might be helpful
>>
>> On thing that came to my mind reading your description, are you sure that
>> the events arrive in the correct order?
>>
>> Best,
>> Fabian
>>
>> [1] https://ci.apache.org/projects/flink/flink-docs-release-1.1/
>> apis/streaming/windows.html
>> [2] http://flink.apache.org/news/2015/12/04/Introducing-windows.html
>>
>> 2016-11-18 1:04 GMT+01:00 Abdul Salam Shaikh <abd.salam.shaikh@gmail.com>
>> :
>>
>>> Hi,
>>>
>>> I have a requirement for my thesis project where I need to set the time
>>> window based on the value which I received in the event stream.
>>>
>>> I need to collect the values when the value starts at 0 and increments
>>> to a max value(which is unknown) till the value resets to 0 again on which
>>> a trigger is triggered which performs a calculation based on the values
>>> collected.
>>>
>>> DataStream<Tuple4<String,Double,String,Integer>> detectorEventStream
=
>>> streamMatchedTuples
>>>         .keyBy(Based on the first param of the tuple)
>>>         .timeWindow(based on the second param of the tuple)
>>>         .apply(new PhaseCalculator());
>>>
>>> I am relatively new in flink so I would like to have ideas on how to
>>> proceed.
>>>
>>> --
>>> Thanks & Regards,
>>>
>>> *Abdul Salam Shaikh*
>>>
>>>
>>
>
>
> --
> Thanks & Regards,
>
> *Abdul Salam Shaikh*
>
>

Mime
View raw message